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The  cost-effectiveness  of  maintenance  simulators,  compared 
to  actual  equipment  trainers,  is  evaluated  for  training  military 
maintenance  technicians.  Maintenance  simulators  are  as  effective 
as  actual  equipment  trainers  when  measured  by  student  achievement 
at  school;  there  is  no  difference  in  the  job  performance  of  stu¬ 
dents  trained  either  way,  according  to  supervisors'  ratings,  in 
only  one  study.  The  acquisition  cost  of  maintenance  simulators 
is  typically  less  than  that  of  actual  equipment  trainers.  The 
cost  to  develop  and  fabricate  one  unit  of  a  simulator  was  less 
than  60  percent  of  the  cost  of  its  counterpart  actual  equipment 
trainer  in  7  of  11  cases  investigated.  The  cost  of  fabricating 
an  additional  unit  of  the  simulator  was  less  than  20  percent  of 
the  cost  of  its  counterpart  actual  equipment  trainer  in  9  of 
these  11  cases.  Acquisition  and  use  of  a  maintenance  simulator 
over  a  15-year  period  would  cost  38  percent  as  much  as  an  actual 
equipment  trainer,  according  to  the  only  life-cycle  cost  compari¬ 
son  that  has  been  reported.  Since  maintenance  simulators  and 
actual  equipment  trainers  are  equally  effective  and  since  main¬ 
tenance  simulators  cost  less,  it  is  concluded  that  maintenance 
simulators  are  cost-effective  compared  to  actual  equipment 
trainers.  This  finding  is  qualified  because  it  is  based  on  a 
limited  number  of  comparisons,  because  effectiveness  is  based 
primarily  on  school  achievement  rather  than  on  the  job  perform¬ 
ance,  and  because  it  is  based  primarily  on  acquisition  rather 
than  on  life-cycle  costs. 
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SUMMARY 


A.  PURPOSE 


This  paper  evaluates  the  cost-effectiveness  of  maintenance 
simulators,  compared  to  actual  equipment  trainers,  for  training 
military  personnel  to  maintain  military  equipment,* 


B .  BACKGROUND 

Maintenance  simulators  are  synthetic  training  devices  that 
appear  to  duplicate  the  performance  characteristics  of  opera¬ 
tional  equipment  under  ncrmel  and  many  malfunction  conditions. 
Actual  equipment  trainers  are  operational  equipments  that  are 
provided  with  power,  inputs,  and  controls  needed  to  make  them 
operate  in  a  classroom.  Maintenance  simulators  incorporate  some 
type  of  computer  support  to  provide  a  large  variety  of  malfunc¬ 
tions  for  instructional  purposes,  are  designed  to  withstand 
abuse  in  a  classroom,  do  not  expose  students  to  dangerous  condi¬ 
tions,  and  can  measure  student  performance  for  the  information 
of  both  students  and  instructors.  They  are  generally  less 
expensive  to  procure  than  actual  equipment  trainers.  Actual 
equipment  trainers  provide  students  .  n  opportunity  to  train  on 
the  actual  equipment  they  will  be  expected  to  maintain  after 


*In  1976,  the  Defense  Science  Board  recommended  cost-effective¬ 
ness  evaluations  of  military  training.  This  study  is  one  of 
several  undertaken  in  response  to  that  recommendation.  The 
study  was  performed  for  the  Office  of  the  Deputy  Under  Secre¬ 
tary  of  Defense  for  Research  and  Engineering  (Research  and 
Advanced  Technology),  under  the  technical  cognizance  of  the 
Military  Assistant  for  Training  and  Personnel  Technology. 
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they  leave  school.  Limitations  of  such  trainers  are  that, 
being  designed  for  operational  rather  than  instructional 
purposes,  they  may  break  down  and  be  difficult  to  maintain  in 
a  classroom  setting.  They  provide  only  limited  opportunities 
for  demonstrating  malfunctions  because  instructors  must  install 
"faulty"  components,  which  always  takes  some  time  and  may  be 
inconvenient.  Actual  equipment  trainers  do  not  include  facili¬ 
ties  for  measuring  student  performance;  this  would  require  a 
complex  and  costly  process  of  redesign. 

C.  EFFECTIVENESS 

The  effectiveness  of  maintenance  simulators  has  been  evalu¬ 
ated  in  12  studies  conducted  since  1967.  These  involved  mainte¬ 
nance  training  for  equipment  used  in  sonar,  avionics,  radar, 
propellers,  flight  control,  navigation,  aircraft  power  plant, 
communication,  and  ship  automatic  boiler  control  systems.  Stu¬ 
dent  achievement  in  12  courses  that  used  maintenance  simulators 
was  the  same  as  or  better  than  that  in  comparable  courses  that 
used  actual  equipment  trainers;  in  one  case,  student  achievement 
with  a  maintenance  simulator  was  less.  In  one  case  where  on-the- 
job  performance  was  evaluated,  supervisors'  ratings  showed  that 
there  was  no  difference  between  students  trained  with  a  simulator 
or  an  actual  equipment  trainer.  Students  trained  with  mainte¬ 
nance  simulators  completed  their  courses  in  less  time  than  did 
those  who  used  actual  equipment  trainers.  In  three  cases  where 
such  data  were  collected,  time  savings  were  22,  50,  and  50  per¬ 
cent,  respectively.  Most  students  who  use  maintenance  simulators 
have  favorable  attitudes  toward  their  use;  instructors  are  split 
about  equally  in  having  favorable,  neutral,  or  negative  attitudes 
toward  the  use  of  these  simulators. 
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D.  COST 


Maintenance  simulators  appear  to  cost  less  to  acquire  than 
do  actual  equipment  trainers.  The  cost  to  design,  develop,  and 
fabricate  one  unit  of  a  simulator  is  less  than  60  percent  of 
the  unit  cost  of  its  counterpart  actual  equipment  trainer  in  7 
cases  out  of  a  sample  of  11;  in  the  remaining  four  cases  the 
simulators  cost  more  than  the  actual  equipment  trainers.  Once 
developed,  the  cost  of  fabricating  an  additional  u">it  of  a 
simulator  is  less  than  20  percent  of  the  unit  cost  of  its 
counterpart  actual  equipment  trainer  in  9  of  those  11  cases; 
in  only  one  case  did  the  simulator  cost  more  to  fabricate  than 
the  actual  equipment  trainer. 

In  the  one  available  case  of  a  life-cycle  cost-effective¬ 
ness  evaluation,  the  Air  Force  6883  Test  Stand  Three-dimensional 
Simulator  was  as  effective  as  the  actual  equipment  trainer,  both 
at  school  and  on  the  job.  The  total  costs  fo*'  the  same  student 
]oad  over  a  15-year  period  were  estimated  to  be  $1.5  million 
for  the  simulator  and  $3.9  million  for  the  actual  equipment 
trainer;  that  is,  the  simulator  would  cost  38  percent  as  much 
to  buy  and  use  as  would  the  actual  equipment  trainer. 


CONCLUSIONS 


Conclusion  1.  Maintenance  simulators  are  as  effective  as 
actual  equipment  trainers  for  training  militay  personnel,  as 
measured  by  students'  achievement  at  school  and,  in  one  case, 
on  the  job.  The  use  of  maintenance  simulators  saves  some  of 
the  time  needed  by  students  to  complete  courses,  but  data  on 
this  point  is  limited.  Students  favor  the  use  of  maintenance 
simulators;  instructors  are  favorable,  neutral,  or  negative 
toward  the  use  of  simulators  in  about  equal  numbers. 

Conclusion  2.  The  acquisition  cost  of  maintenance  simula¬ 
tors  are  typically  less  than  that  of  actual  equipment  trainers. 
The  cost  to  develop  and  fabricate  one  unit  of  a  simulator  was 
less  than  60  percent  of  the  cost  of  actual  equipment  trainers 
in  7  of  11  cases  examined;  the  cost  to  fabricate  an  additional 
unit  of  a  simulator  was  less  than  the  20  percent  of  cost  of 
actual  equipment  trainers  in  9  of  the  11  cases.  The  one  avail¬ 
able  life-cycle  cost  estimate  shows  that  purchase  and  use  of  a 
simulator  would  cost  38  percent  as  much  over  a  15-year  period 
as  it  would  to  buy  and  use  an  actual  equipment  trainer. 

Conclusion  3.  Maintenance  simulators  are  as  effective  as 
actual  equipment  trainers  for  training  maintenance  personnel. 

In  addition,  they  cost  less  to  acquire.  Therefore,  maintenance 
simulators  are  cost-effective  when  compared  with  actual  equip¬ 
ment  trainers. 

Conclusion  4.  In  general,  the  data  on  the  cost  and  effec¬ 
tiveness  of  maintenance  simulators  have  not  been  collected  sys¬ 
tematically.  Therefore,  there  is  no  basis  at  present  for  mak¬ 
ing  trade-offs  between  the  effectiveness  and  cost  of  different 
types  of  maintenance  simulators  on  such  issues  as  two-dimensional 


versus  three-dimensional  design,  the  complexity  of  maintenance 
simulators  (in  such  terms  as  number  of  malfunctions  and  instruc¬ 
tional  procedures) ,  the  extent  to  which  simulators  should  pro¬ 
vide  a  mixture  of  training  in  general  maintenance  procedures 
and/or  for  maintaining  specific  equipments,  and  the  optimum 
combination  of  maintenance  simulators  and  actual  equipment 
trainers  for  training  technicians  at  school. 

There  have  been  insufficient  studies  on  the  amount  of  stu¬ 
dent  time  saved  with  the  use  of  maintenance  simulators.  There 
have  been  no  studies  on  whether  the  use  of  maintenance  simula¬ 
tors  influences  the  amount  of  student  attrition  at  school. 

There  have  teen  no  studies  to  collect  objective  measures  of 
performance  of  maintenance  technicians  on  the  job  after  train¬ 
ing,  either  with  simulators  or  actual  equipment  trainers. 

Conclusion  5.  Maintenance  simulators  now  under  development 
have  not  yet  taken  advantage  of  recent  technological  advances 
such  as  videodiscs,  automated  voice  input  and  output,  and  minia¬ 
turization  sufficient  to  make  them  readily  portable.  Reductions 
in  size  would  make  it  possible,  as  well  as  more  convenient,  to 
use  maintenance  simulators  for  refresher  training  near  job  sites 
and  for  performance  evaluation  and/or  certification  of  mainte¬ 
nance  personnel  on  an  objective  basis  in  operational  environ¬ 
ments.  Extreme  reductions  in  size  would  make  it  possible  to 
use  maintenance  simulators  as  job  aids  in  performing  maintenance 
on  operational  equipment,  thus  assuring  a  close  link,  not  yet 
available,  between  facilities  used  for  training  at  school  and 
for  performance  on  the  job. 
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RECOMMENDATIONS 


Recommendation  1.  Collect  data  to  enlarge  what  is  now 
known  about  the  effectiveness  of  maintenance  simulators  and 
actual  equipment  trainers  at  school.  Data  are  needed  on  speci¬ 
fic  knowledge  and  skills  acquired  by  students  at  school,  the 
time  needed  to  complete  courses,  attrition  rates,  and  instruc¬ 
tor  attitudes  toward  the  use  of  simulators  and  actual  equipment 
trainers. 

Recommendation  2.  Collect  objective  data  on  the  performance 
of  technicians  on  the  job  after  training  with  simulators  or 
actual  equipment  trainers.  Determine  the  transfer  of  training 
of  maintenance  skills  from  school  to  the  job,  when  either 
maintenance  simulators  or  actual  equipment  trainers  are  used 
in  training  courses.  Such  data  should  be  collected  in  a  way 
that  will  permit  a  determination  of  the  relative  effectiveness 
of  maintenance  simulators  with  varying  characteristics  such  as, 
types  of  design,  degrees  of  complexity,  physical  appearance,  and 
in  generic  and  specific  maintenance  training  courses. 

Recommendation  3.  Collect  cost  data  in  sufficient  detail 
to  permit  the  development  of  cost-eetimating  relationships  for 
maintenance  simulators.  The  cost  elements  should  account  for 
all  portions  of  the  total  costs  incurred  to  procure  and  use 
maintenance  simulators  and  actual  equipment.  A  suggested  struc¬ 
ture  for  the  collection  of  procurement  cost  data  is  contained  in 
this  paper. 

Recommendation  4.  Design  and  conduct  studies  of  training 
with  maintenance  simulators  and  actual  equipment  trainers, 
that  will  yield  trade-offs  between  level  of  effectiveness 
and  total  cost  as  functions  of  the  ch  .  icteristics  of  training 
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equipment,  the  ways  it  is  used,  and  the  types  of  training 
involved. 

Recommendation  5.  Develop  a  procedure  to  categorize  the 
functional  characteristics  of  maintenance  simulators  and  actual 
equipment  trainers  in  ways  that  will  relate  to  their  effective¬ 
ness  for  training. 

Recommendation  6  Develop  objective  measures  of  the  job 
performance  of  maintenance  personnel  in  operational  settings  to 
provide  valid  measures  with  which  to  evaluate  the  effectiveness 
of  simulators  and  actual  equipment  trainers. 
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I.  INTRODUCTION 


A.  PURPOSE 

The  purpose  of  this  paper  is  to  evaluate  the  cost-effec¬ 
tiveness  of  maintenance  training  simulators,  compared  to  actual 
equipment  trainers,  for  use  in  training  military  personnel  to 
maintain  operational  equipment.  Both  types  of  equipment  have 
been  used  at  technical  training  schools  to  trair.  personnel  to 
perform  corrective  and  preventive  maintenance  at  organizational 
and  intermediate  levels. 

Actual  equipment  trainers  have  long  been  used  in  technical 
training  schools  for  two  significant  reasons:  (1)  they  can  be 
acquired  simply  by  ordering  additional  units  of  operational 
equipment  already  being  procured  for  use  in  weapon  and  support 
systems;  and  (2)  they  provide  realistic  training  on  the  equip¬ 
ment  to  be  maintained  after  the  student  leaves  school.  Opera¬ 
tional  equipment  is  often  modified  for  training  purposes  by, 
for  example,  placing  it  on  a  stand  and  adding  power  supplies, 
input  signals,  and  controls  needed  to  make  it  operate  in  a 
classroom.  There  has  been  a  trend,  in  recent  years,  to  use 
maintenance  training  simulators  rather  than  actual  equipment 
for  training  purposes.  Maintenance  simulators  are  said  to  have 
advantages  for  use  in  training  such  as  lower  cost,  ability  to 
demonstrate  a  wider  variety  of  malfunctions,  and  more  freedom 
from  breakdown  in  the  classroom.  These  advantages  will  be 
considered  later  in  this  paper. 

The  purpose  of  maintenance  training  is,  obviously,  to 
train  personnel  to  maintain  complex  equipment;  this  requires 
both  technical  knowledge  and  manual  job  skills.  Maintenance 
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training  familiarizes  the  student  with  the  layout  of  the  equip¬ 
ment,  sources  of  power,  use  of  tools  and  test  equipment,  safety 
requirements,  control  settings,  instrument  readings,  operating 
procedures,  and  the  like.  Maintenance  personnel  must  be  able 
to  diagnose  malfunctions;  identify,  replace,  or  repair  faulty 
components;  verify  that  all  components  perform  within  prescribed 
tolerances;  and  perform  tests  to  insure  that  the  entire  equip¬ 
ment  has  been  returned  to  working  order.  This  type  of  training 
can  be  provided  by  a  variety  of  mean;,  such  as  conventional  class¬ 
room  instruction,  studying  technical  manuals,  learning  fault¬ 
finding  procedures  by  self-study,  computer-assisted  or  computer- 
managed  instruction  and,  of  course,  the  use  of  various  types  of 
training  devices.  The  issue  addressed  in  this  paper  is  whether 
maintenance  simulators  are  more  costef fective  than  actual  equip¬ 
ment  trainers  for  training  military  maintenance  personnel. 

Even  if  maintenance  simulators  are  more  cos t-ef feci tve  at 
school  for  training  personnel,  it  is  obvious  that  training  is 
supposed  to  prepare  technicians  to  maintain  operational  equip¬ 
ment  in  the  field  and  not  just  to  perform  well  at  school.  Thus, 
the  major  substantive  issue  is  to  compare  how  personnel  trained 
with  maintenance  simulators  or  actual  equipment  trainers  actually 
maintain  operational  equipment  in  the  field.  Whether  this  ques¬ 
tion  can  be  answered  on  the  basis  of  currently  available  infor¬ 
mation  is  considered  later. 

D.  TYPES  AND  LOCATIONS  OF  MAINTENANCE  ACTIVITIES 

There  are  two  main  types  of  maintenance  activities: 

1.  Corrective  maintenance  applies  to  equipment  that  has 
failed  or  is  known  to  be  operating  improperly.  In  the 
typical  case,  a  malfunction  is  noted  and  reported  by 
operational  personnel  who  use  the  equipment  and  re¬ 
paired  by  the  maintenance  personnel.  Corrective 
maintenance  involves  troubleshooting,  diagnosing  the 


1 


» 


9 


9 


10 


reason  for  a  malfunction,  identifying  the  component 
{or  components)  that  has  failed,  repairing  and/or 
replacing  the  faulty  component  (or  larger  module  of 
which  it  may  be  a  part)  and,  finally,  testing  and 
calibrating  to  assure  that  the  malfunction  no  longer 
exists . 

2.  Preventive  maintenance  applies  to  equipment  that  has 
not  failed  and  appears  to  be  operating  properly.  It 
involves  periodic  inspection,  cleaning,  testing,  and 
calibrating  of  equipment;  this  may  include  the  replace¬ 
ment  of  functioning  parts  in  accord  v/ith  schedules 
established  to  reduce  the  possibility  of  future  break¬ 
downs  . 

To  be  effective,  both  types  of  maintenance  require  not 
only  proper  training  but  also  proper  tools,  test  equipment, 
relevant  and  up-to-date  technical  documentation,  and  efficient 
diagnostic  procedures;  the  equipment  itself  must  be  designed 
to  permit  convenient  access,  test,  repair,  ar.6  replacement  of 
parts;  and  there  must  also  be  a  proper  supply  of  spare  parts 
and  an  adequate  number  of  maintenance  personnel,  including 
supervisors,  to  handle  the  workload. 

Maintenance  activities  are  also  associated  with  the  places 
where  they  occur.  There  are  three  types,  as  described  below: 

1.  Organizational  maintenance  is  performed  on  equipment 
on  the  flight  line  or  in  the  field  by  maintenance 
personnel  assigned  to  the  unit  that  operates  the  equip¬ 
ment.  It  consists  generally  of  inspecting,  servicing, 
lubricating,  adjusting,  and  replacing  faulty  assemblies 
and  subassemblies  (line-replaceable  units  or  LRUs). 

2.  Intermediate  maintenance  is  performed  in  maintenance 
shops  by  personnel  assigned  to  a  base  or  support  organi¬ 
zation.  It  generally  consists  of  calibration,  repair 

or  replacement  of  damaged  or  unserviceable  parts,  the 
emergency  manufacture  of  nonavailable  parts,  and  pro¬ 
viding  technical  assistance  to  the  using  organization. 
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3.  Depot  maintenance  is  performed  at  a  central,  industrial- 
type  facility  and  consists  of  large-scale  repair,  modi¬ 
fication,  and  refurbishment.  t 

C.  MAGNITUDE  OF  THE  PROBLEM 

The  Department  of  Defense,  as  of  1976,  possessed  weapon  * 

systems  and  equipment  which  cost  about  $125  billion  to  acquire 
(see  Table  1).  About  $49  billion  was  requested  for  procurement 
in  FY  1982  (Brown  1981,  p.  312).  The  purpose  of  maintenance  is 
to  keep  these  weapons  and  their  support  equipment  in  a  state  * 

of  operational  readiness  to  meet  mission  requirements  and  to  do 
this  in  a  timely  and  economic  manner.  Maintenance  is  a  criti¬ 
cal  aspect  of  defense  planning  and  operations  and  costs  $18-20 
billion  each  year,  including  the  costs  of  spare  parts,  supplies,  , 

and  modifications  (Turke  1977,  p.  5). 

TABLE  1.  ACQUISITION  COST  OF  WEAPON  SYSTEMS  AND  EQUIPMENT  IN 
USE  OR  ASSIGNED,  JULY  1,  1976  (TURKE,  1977) 
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Acquisition  Cost 

Billions  of 
Dollars 

Percent 

Military  Department 

Army 

19.2 

15 

Navy 

61.7 

49 

Air  Force 

45.3 

36 

126.2 

100 

Weapons  Group 

Aircraft 

54.1 

43 

Ships 

38.8 

31 

Missiles 

8.7 

7 

Vehicles 

9.0 

7 

Other 

15.6 

12 

126.2 

100 
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According  to  the  General  Accounting  Office,  the  Army  spends 
25  percent  ($7.0  billion  in  FY  1978)  of  its  annual  budget  on 
maintenance;  over  200,000  mechanics  and  equipment  operators  in 
the  Army  have  specific  unit-level  maintenance  responsibilities 
(GAO  1978,  p.  1).  In  the  Air  Force,  maintenance  requires  about 
28  percent  of  the  work  force  (military  and  civilian)  and  costs 
between  $5  and  $7  billion  annually  (Townsend  1980).  Labor  for 
repairs  is  estimated  to  account  for  39  percent  of  the  cost  of 
recurring  logistical  support  (Fiorello  1975).  Training  is  only 
one  of  many  factors  that  influence  effective  maintenance,  e.g., 
design  of  equipment  to  assure  high,  inherent  reliability;  design 
of  equipment  to  permit  unambiguous  identification  of  failed 
components;  easy  access  for  test  and  replacement  of  components; 
the  availability  of  spare  parts  and  test  equipment;  up-to-date 
technical  documentation,  tools,  job  aids,  and  the  like.* 
Specialized  skill  training  at  military  schc  1 s  costs  about 
$2.9  billion  or  33  percent  of  the  cost  of  inuividual  training 
each  year  [Department  of  Defense,  Military  Manpower  Training 
Report  ( MMTR)  for  FY  1981,  i  .  C 1  ;  the  portion  at'-ributed  solely 
to  maintenance  training  is  not  known. 

High  turnover  among  enlisted  personnel  increases  the  diffi¬ 
culty  of  maintaining  military  equipment.  According  to  planning 
estimates  for  FY  1981,  about  337,000  personnel  were  to  be  re¬ 
cruited;  313,000  (93  percent)  of  these  were  expected  to  complete 
recruit  training;  only  64,000  (37  percent)  would  reenlist  for 
a  second  term  [MMTR  FY  1981,  p.  III-3;  estimate  on  reenlistment 
from  all  volunteer  force  data  base,  ASD  (MRA&L),  20  Mar  19801. 

*See  Integrated  Technical  Documentation  and  Training  ( I TDT) 
(1978)  and  Navy  Technical  Information  Presentation  Proqram 
(NTIPP)  (1977)  for  a  review  of  current  efforts  to  improve 
technical  documentation  required  for  maintenance;  see  Rowan 
(1973)  and  Post  and  Price  (1973)  for  recent  reviews  of  studies 
which  compare  performance  of  maintenance  technicians  using 
innovative  performance  aids  or  conventional  documentation. 
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About  393,000  enlisted  personnel  were  expected  to  complete 
initial  skill  training  courses  (after  8  percent  attrition)  and 
160,000  to  complete  skill  progression  training  (after  5  percent 
attrition)  (MMTR  FY  1981,  p.  V-4,  V-7 ) .  The  ccsts  of  on-the-job 
training  which  follows  school  training  are  essentially  unknown 
(they  are  included  among  the  costs  for  Operation  and  Maintenance 
which  are  $62.4  billion  in  FY  1982  (Brown  1981,  p.  312). 

The  three  Services  spent  over  $5  million  in  FY  1979  for  re¬ 
search  and  development  on  maintenance  simulators;  this  amount  is 
projected  to  decrease  to  about  $1.6  million  by  FY  1983  (Table  2) 
About  $3.7  million  (68  percent)  of  the  FY  1979  funds  (category 
6.4  funds)  were  for  the  development  and  procurement  of  prototype 
equipment;  about  49  percent  of  all  funds  for  FY  1980-1983  would 
also  be  allocated  to  prototype  equipment.  Maintenance  simula¬ 
tors  either  under  contract  or  planned  for  development,  as  of 
February  1981,  are  listed  in  Table  3. 

Over  a  7-year  period  (FY  1975-1981),  the  Naval  Training 
Equipment  Center  alone  procured  training  equipment  at  a  cost  of 
$649  million;  planned  procurements  as  of  March  1980  were  for 
an  additional  $305  million.  Maintenance  trainers  will  account 
for  $3.2  million  or  0.3  percent  of  these  procurements;  equip¬ 
ment  with  a  unit  cost  less  than  $100,000  is  not  included  in 
these  figures  (private  correspondence,  NTEC  N-7,  8  March  1980). 

The  Air  Force  Air  Training  Command  estimates  that  the  cur¬ 
rent  inventory  of  all  maintenance  training  devices  cost  $500 
million,  of  which  $350  million  is  for  aircraft  maintenance  alone 
(Aeronautical  Systems  Division,  1978).  There  are  thought  now 
to  be  about  3600  different  types  of  maintenance  training  devices 
in  the  inventory  to  support  aircraft  systems.  The  procurement 
of  maintenance  simulators  for  the  F-16  aircraft  is  estimated 
to  cost  about  $32  million  (this  includes  some  units  to  be 
delivered  to  NATO  countries). 


TABLE  2.  FUNDS  FOR  R&D  ON  MAINTENANCE  SIMULATORS  BY  MILITARY  SERVICES 

FY  1978  -  1983 

Funds  'thousands  of  dollars) 


Service/PE 

"Ty  78 
and 
Prior 

FY  79 

FY  80 

FY  81 

FY  82 

FY  83 

Army 

6.2 

47 

600 

800a 

6503 

6003 

ttS 

O 

o 

cr> 

6.4 

2000 

2000 

2000 

-- 

-- 

-- 

(Total ) 

(2047) 

.  (2600). 

(2800) 

_  (650) 

(6001 

(900) 

Navy 

6.3 

2723 

362 

516b 

1 2lOb 

H25b 

6.4 

1703 

1665 

233C 

275C 

302C 

211 

(Total) 

(4426) 

(2027) 

.(7.49L 

(1485) 

(1427) 

(211) 

Air  Force 

6.3 

600 

800 

640d 

00 

o 

o 

CL 

700d 

500d 

(Total) 

(60 0) 

(800) 

(640) 

(800) 

(700) 

(500) 

TOTAL 

7073 

5427 

4189 

2935 

2727 

1611 

aPE  62727A-230  BO:  AMTESS  II,  Software,  BITE/AMTE.  PM  TRADE  FY  1981 
Apportionment  Review,  10  June  1980. 

bPE  63733N  W  1202-PN  IMTS;  W  1201-PN  I  HOMS ;  W  1207-PN  ATE.  NTEC 
R&D  Program,  February  1980. 

CPE  64703N  W  0784-PN  SAMT.  NTEC  R&D  Program,  February  1980. 

dPE  63751 F  2351,  6833,  Flat  panel  simulator.  FY  1981  AFHRL 
Apportionment  Review  Data  Book,  27  June  1980. 

Source:  Joint  Technical  Coordinating  Group  -  Sub-group  for  Main¬ 
tenance  Simulators,  December  1978  (Draft),  with  modifica¬ 
tions  noted  above. 
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TABLE  3.  MAINTENANCE  TRAINING  SIMULATORS  UNDER  CONTRACT 
FOR  PRODUCTION  OR  DEVELOPMENT  OR  PLANNED  FOR  DEVELOPMENT 
(as  of  February  1981) 


System  Slmuloted  or 
Simulator  Design  on 

Service3 

Maintenance 

Echelon. 

Trained6 

Type  of 
Simulator^ 

Devlce/Proqram 

Characterlstlscs  and  Status 

AMTESS  (Army  Maintenance 
Training  and  Evaluation 
Simulation  System) 

A 

2-0/3-D 

To  provide  Initial  training  In  di¬ 
verse  skill  areas.  Contract  lei 

In  December  of  1980  for  delivery 
in  October  of  i  J  of  two  "bread¬ 
board"  units  for  evaluation. 

EEMT  (Electrical/Elec¬ 
tronic  Maintenance 
Trainer) 

N 

2-D/3-0 

Initial  (Navy  A-school )  training 
for  electronic  and  electronic  war¬ 
fare  technician  ratings.  Con¬ 
tracts  awarded  In  June  and  July 
of  1980  for  delivery  of  twenty 
2-dlmenslonal  and  two  3-dlmenslon- 
al  prototype  units. 

Fire  Control/Search  Radar 
Maintenance  Trainer 

N 

Initial  (Navy  A-school )  training 
for  fire  control  technician 
rating.  Front-end  analysis  com¬ 
pleted.  RFP  planned  for  release 

In  March  with  contract  award  for 
units  anticipated  by  August  1981. 

bE  and  1 1  Hb  7/ 

Terles  Trainers 

N 

2-D 

Small  flat  panel  devices  for 
basic  skill  training  (Navy  A- 
school)  In  several  skill  areas. 

Contract  awards  to  two  firms  en¬ 
compass  20  simulations  ind  194 
trainers.  Deliveries  on  one  con¬ 
tract  are  scheduled  to  be  com¬ 
pleted  In  February  1981 . 

Deliveries  on  the  second  contract 
are  scheduled  to  begin  In  the 
spring  of  1981  and  to  be  com¬ 
pleted  In  December. 

ROLAND  Institutional 

T  ra 1 ner 

A 

0  and  I 

2-0/ 3-D 

Training  In  electronic  and  hy¬ 
draulic  systems  at  organization¬ 
al  and  direct  support  (DS)  eche¬ 
lons.  RFP  released  In  December 

1980,  contract  award  planned  for 

October  1981.  Contract  Is  to 

Include  five  organizational 
trainers,  two  DS  echelon 
trainers,  and  two  mockups. 

FIREFINDER 

A 

0  and  1 

2-D 

Provides  operator  and  mainten¬ 
ance  (organizational  and 
intermediate  echelons)  training 
for  mortar-  and  artlilery- 
locatlng  radars.  Contract  award¬ 
ed  August  1977  for  36  trainers 
for  operator  training  of  both 
radars  and  maintenance  training 
of  the  mortar-locating  radar; 
deliveries  began  in  January 

1980  and  should  be  completed  in 
early  1981.  Maintenance  train¬ 
er  for  artillery-locating 
radar  will  be  developed/pro¬ 
cured  on  a  subsequent  contract. 

(See  Randle  1980). 

_ 

(Continued) 
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System  Simulated  or 
Simulates  Designation 

C  3 

Service 

Maintenance 

Echelon. 

Trained0 

Type  of 
Simulatoi 

Device/Program 

Characteristics  and  Status 

REES  (Reactive  Electronic 
Equipment  Simulator) 

A 

0 

3-D 

Provides  operator  and  organ¬ 
izational  maintenance  training 
for  the  Army  Tactical  Communi¬ 
cations  System  (ATACS).  Contract 
awarded  In  September  1977  for 
one  4-statlon  network.  Delivery 
currently  anticipated  for  mid  to 
late  Summer  1981. 

FVS  (Fighting  Vehicle 
System)  Maintenance 
Trainer 

A 

0  and  1 

2-0 

Contract  has  been  let  for  design 
and  data.  Contract  for  fabric¬ 
ation  planned  to  be  funded  from 

FY  1982  budget  and  to  Include 
four  different  simulations  and 
a  hands-on  trainer,  all  for 
turret  maintenance. 

XM-1  Tank  Maintenance 
Training  System 

A 

0 

2-0 

In  procurement;  Includes  six 
simulations  (covering  five  tank 
subsystems)  and  hands-on 
trainer.  Delivery  of  prototypes 

Is  scheduled  to  tegln  In  Feb¬ 
ruary  1931.  Delivery  of  produc¬ 
tion  versions  Is  scheduled  for 

July  1982  to  February  1984. 

M109/1 10  Turret  Trainer 

A 

0 

2-D 

In  procurement;  deliveries  to 
begin  In  July  '981.  One  sim¬ 
ulation  provides  training  1 n 
electrical  and  hydraulic  main¬ 
tenance  for  self-propelled 
artillery. 

IRR  (Integrated  Radio 

Room) 

N 

0 

3-D 

Provides  operator  and  mainten¬ 
ance  training  of  Trident  sub¬ 
marine  communications  system. 

One  system  (consisting  of  a 
simulated  communications  system 
and  several  part-task-trainers) 
was  placed  under  contract  In 

September  1979  and  Is  scheduled 
for  delivery  In  March  1981.  A 
seccnd  system  may  be  procured. 

Mk  92  FCS  (Fire  Control 
System 

N 

0 

2-D 

Currently  on  letter  contract  (to 
be  made  definite  In  February  or 

March  1981),  with  first  deliver¬ 
ies  scheduled  for  March  1982. 
Configuration  Is  a  modification 
of  the  TICCIT  system  that  Inte¬ 
grates  conventional  flat  panels 
in  a  12-student-statlon  TICCIT 
complex.  Contract  will  Involve 
two  complexes. 

C I WS  (Phalanx  Close-In 
Weapon  System) 

N 

0 

2-D 

Letter  contract  (signed  In  Oct¬ 
ober  1980)  to  be  made  definite 

In  February  1981  with  deliveries 
scheduled  to  begin  In  November 

1981.  Contract  provisions 
specify  delivery  of  39  sets  of 

(Continued) 

> 


17 


TABLE  3 
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System  Simulated  or 
Simulator  Designation 

Service3 

Maintenance 

Echelon 

Trained*3 

Type  Of 
Simulator 

Device/Program 

Characteristics  and  Status 

eight  simulations  (panels  por¬ 
traying  different  subsystems) 
and  one  3-dlmenslonal  model. 

DD  963  Waste  Heat  Boiler 

N 

0 

Contracted  In  December  1980  for 
delivery  of  one  set  of  three 
simulations  In  December  1981. 
firm  design  will  not  be  set  un- 
tl 1  March  or  Aprl 1  1981 . 

300-89  Sonar  Maintenance 
Trainer  (formerly  the 
Generalized  Sonar 
Maintenance  Trainer) 

N 

Large  program  consisting  of  op¬ 
eration  and  maintenance  trainers 
for  three  sonar  systems.  Com¬ 
plete  program  planned  to  consist 
of  both  simulation  and  stimulated 
operational  equipment.  RFP  for 
procurement  of  operator  trainers 
scheduled  to  be  Issued  In  March 

1981,  with  resulting  contract  to 
be  funded  from  FY  1981  budget. 
Maintenance  trainer  front-end 
studies  to  be  contracted  from 

FY  1981  funds  with  procurement 
planned  from  FY  1983  budget. 

Hagen  ABC  (Automatic 

Boiler  Control) 

N 

0 

2-0/3-D 

Prior  contracts  resulted  In  pro¬ 
curement  and  evaluation  of  three 
units  employed  for  research. 

Current  funding  Is  to  modify  the 
three  simulators  to  the  current 
configuration  of  the  operational 
equipment  for  use  In  mainstream 
training. 

Woodward  Govern. r 

N 

0 

Research  program.  Front-end 
analysis  essentially  completed. 

Current  funding  provides  for 
design/development  of  audio-visual 
and  courseware  (other  than  EDP) 
materials.  Design/development  of 
hardware  and  EDP  software/course¬ 
ware  to  be  Initiated  with  future 
year's  funding. 

A6-E  tram  ors  (Dectectlon 
and  Ranging  5ystem) 

N 

0 

3-D 

Two  trainers  delivered  under  prior 
contracts.  Current  funding  Is 
limited  to  updating  these  devices 
to  the  current  configuration  of 
the  operational  equipment. 

CA-6B  t CAP  - 1  TJS 
|  1 ’  ictl ca  1  Jamml ng 

system) 

N 

0 

3-D 

Procurement  contract  awarded  In 

December  1980  for  two  units.  Deliv¬ 
eries  are  scheduled  for  January  and 
March  1982.  Current  planning  In¬ 
cludes  later  modification  of  at 
least  one  unit  to  the  I CAP - 1 1  air¬ 
craft  configuration. 

1 _ 

(  ontlnued) 
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System  Simulated  or 
Simulator  Designation 

Service® 

Maintenance 

Echelon. 

Trained0 

Type  of 
Simulator 

Device/Program 

Characteristics  and  Status 

AT  Trainer 

N 

0 

3-D 

Military  In-house  program  Intlated 

In  December  1975.  Delivery  of 
partially  configured  trainers  In 

June  1979  (1  device)  and  December 

1980  (Z  devices).  Modification 
of  delivered  trainers  to  full 
design  configuration  Is  scheduled 
for  completion  In  November  1981. 

MA-3  Test  Bench/CSD 
(Constant  Speed  Drive) 

N 

1 

3-D 

Research  program.  Single  device 
delivered  In  August  1980.  Current 
funding  provides  support  for  on¬ 
site  training  evaluations. 

SH-3  Blade- fold  Trainer 

N 

0 

Front-end  analysis  scheduled  for 
completion  during  Summer  1981.  No 
decisions  regarding  the  program 
are  anticipated  before  that  time. 

F- 18  Maintenance  Trainers 

N 

Front-end  study  rccoemended  a  mix 
of  simulations  and  operational 
equipment  trainers.  RFPs  for  five 
simulators  were  released  In  Jan¬ 
uary  1931.  Contract  awards  are 
expected  to  begin  In  April  1981. 
Procurements  will  be  managed  by 
McDonnell  Douglas  as  sub-contracts 
tn  the  basic  weapon  contract*. 

6883  Test  Bench 

(Flat  Panel  trainer) 

AF 

1 

Z-D 

Research  program.  Single  device 
delivered  <n  August  1980.  FY 

1981  funding  Is  to  support  an  on¬ 
site  training  evaluation. 

T -16  Maintenance 

Simulators 

AF 

0 

Z-D/3-D 

Initial  contract  (September  1977) 
provided  for  delivery  of  six  sets 
of  18  simulations.  Deliveries  of 
six  sets  of  1Z  modified  (degraded) 
simulations  were  completed  by 

September  1980,  FV  1981  and 
later  funding  Is  to  provide  for 
retrofit  of  delivered  articles  to 
their  Initial  design  configura¬ 
tion  and  productlon/del 1  very  of 
the  remaining  simulations. 

C-3A  (AWACS)  Radar 

AF 

0 

Z-D 

In  development.  Contract  award¬ 
ed  In  September  1980  for  one 
simulator  containing  10  student 
stations.  el Ivery  anticipated 

In  May  19bZ. 

E-3A  (AWACS)  Data 
Olsplay/Control  System 

AF 

0 

Front-end  analysis  nearing  com¬ 
pletion.  RFP  scheduled  for 
release  In  March  1981,  Contract 
award  anticipated  at  the  end  of 

FY  1981. 

(Continued) 
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System  Simulated  or 
Simulator  Designation 

Service3 

Maintenance 

Echelon. 

Trained0 

Type  of 
Simulator0 

Device/Program 

Characteristics  and  Status 

EA-6B  I CAP-- 1 1  CNR 
(Communication/ 
Navigation) 

N 

0 

Front-end  analysis  and  detailed 
specification  scheduled  for  com¬ 
pletion  in  April  1981,  Funding 
for  procurement  anticipated  from 
the  FT  1982  budget. 

E-3A  (AWACS)  Advanced 
Radar  Maintenance 
Trainer 

AF 

0 

For  training  to  advanced  skill 
level.  Functional  requirements 
study  in  initial  stages.  Con¬ 
tract  award  not  anticipated 
before  the  end  of  FY  1982. 

AN/TPS-43E  Radar 

AF 

0 

3-D 

Military  in-house  project.  Pro¬ 
gram  initiated  in  early  19?7. 

First  device  placed  in  use  In 
late  1978.  A  second  unit  is 
currently  being  fabricated. 

JA:  Army;  N:  Navy,  AF :  Air  force, 

I:  Intermediate-level  ma intenance ;  0:  Oraqnl za clonal  -  level  maintenance, 
cTwo-d1mens1onal ,  three-dimensional . 

^Proposed  for  M  109/110  howitzer  turret,  M60  tank,  M809  truck,  radar  illuminator, 
eAlsi)  called  the  Simulated  Avionics  Maintenance  Trainer. 
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One  large  industrial  contractor  has  estimated  that  the 
Department  of  Defense  will  speiid  about  $620  million  for  mainte¬ 
nance  trainers  over  the  period  1977-1985;  annual  procurements 
are  estimated  to  reach  about  $120  million  per  year  by  1985 
(Pig.  1).  The  distribution  of  this  procurement,  according  to 
type  of  trainer,  is  predicted  to  be  as  shown  in  Fig.  2.  Out¬ 
side  the  United  States,  the  procurement  of  maintenance  simu¬ 
lators  is  estimated  to  be  about  $5.5  million  per  year. 


FIGURE  1.  Estimated  procurement  of  maintenance 

trainers  by  the  Department  of  Defense, 
1975-1985  (as  of  November  1979) 
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$424  M 

71 


42 

38 

34 

12 

$621  M 


FIGURE  2.  Predicted  procurement  of  maintenance  trainers 
by  the  Department  of  Defense,  according  to 
*  pe  of  application,  1977-1985  (estimate 
mfele  In  November  1979) 


The  "El ectroo ics-X"  study,  conducted  in  1974,  was  a  major 
effort  to  determine  the  cost  and  reliability  of  military  elec¬ 
tronic  equipment  (Gates,  Gourary,  Deitchman,  et  al.,  1974), 

Four  methods  were  used  to  estimate  the  cost  of  maintaining 
electronics  equipment  each  year.  The  results  ranged  from  $3.4 
billion  to  $6.8  billion,  with  an  average  of  $5.4  billion  per 
year  (Gates,  Gourary,  Deitchman  et  al.,  1974,  Vol.  II,  p.  374). 
The  estimate  of  $5.4  billion  per  year  for  maintenance  is  about 
equal  to  the  cost  of  procuring  electronic  equipment  each  year 
(Gates,  Gourary,  Deitchman  et  al.,  1974,  Vol.  I,  p.  52).  Note 
that  procurement  costs  relate  to  acquiring  current  technology; 
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the  maintenance  costs  relate  to  systems  whose  average  age  is 
about  10  years. 

Advanced  military  equipment  has  become  more  complex  in  order 
to  provide  improved  performance.  Increased  complexity  brought 
increased  cost  and  decreased  reliability,  the  latter  imposing 
increased  demands  on  maintenance  personnel  and  resources.  The 
Electronics-X  study  showed  that  the  reliability  of  avionics 
equipment  in  the  field  decreases  with  increases  in  unit  cost 
for  aircraft  in  accordance  with  the  following  relationship: 

Aircraft  MFHBF*  =  1.3  x  106/cost  . 

As  shown  in  Figure  3.  more  expensive  (and  more  complex)  elec¬ 
tronics  equipment  has  a  lower  reliability  and  creates  a  larger 
demand  on  maintenance  activities  than  does  less  expensive 
equipment  (Gates,  Gourary,  Deitchman,  et  al.  1974,  Vol.  I,  p. 

56).  A  similar  relationship,  based  on  limited  data,  was  found 
for  Army  Area  Communications  Systems  (AACS)  where 

AACS  MTBF**  =  107/cost  . 

The  costs  for  manpower  were  estimated  by  a  Defense  Science 
Board  (DSB)  Task  Force  on  Electronics  Management  to  account 
for  perhaps  as  much  as  75  percent  of  the  military  electronics 
maintenance  costs;  actual  costs  are  unknown  due  to  limitations 
in  the  cost  allocation  system  (DSB,  1974,  p.  14). 


*Mean  flight  hours  between  failures. 
**Mea.i  time  between  failures. 


100  200  500  1000  2000  5000 


ii  in*  it 

FIGURE  3 


20,000  100,000 

10,000  50,000  200,001 

UNIT  PRODUCTION  COST  (dollars) 


Avionics  field  reliability  versus 
uni t  producti on  cost 


D.  TYPES  OF  MAINTENANCE  SIMULATORS 


Maintenance  simulators  now  under  development  differ  notably 
in  their  resemblance  to  actual  equipment,  their  functional  capa¬ 
bilities  as  instructional  devices,  and  in  their  complexity  and 
cost.  Modern  maintenance  simulators  are  often  characterized 
as  2-D  or  3-D  devices,  i.e.,  as  being  two-  or  three-dimensional 
in  their  physical  form;  some  simulators  contain  both  2-D  and 
3-D  components. 

The  2-D  devices  consist  of  flat  panels  with  drawings  of 
major  components  connected  symbolically  by  flow  diagrams  to 
show  electrical  and/or  hydraulic  functional  connections  be¬ 
tween  components.  The  panels  contain  functioning  instruments, 
signal  lights,  and  controls,  so  that  the  technician  can  turn  on 
power  to  the  equipment,  see  if  it  is  working  correctly,  and 
observe  the  effects  of  various  actions  he  may  take  to  identify 
and  correct  the  malfunctions  that  are  present.  Such  panels 
perform  as  if  they  were  real  equipment  because  each  contains 
a  computer,  with  a  mathematical  model  of  the  real  system  that 
makes  the  displays  respond  appropriately  to  all  settings  of 
the  controls  under  all  environmental  conditions  likely  to  be 
encountered.  By  setting  a  switch  on  his  panel,  the  instructor 
can  select  a  malfunction  from  a  large  set  contained  in  the 
computer.  The  equipment  scores  the  student's  performance  and 
tells  him  whether  he  has  correctly  identified  a  malfunction. 

The  instructor  can  stop  the  sequence  of  activities  for  instruc¬ 
tional  purposes,  to  reoeat  what  the  student  has  done,  and 
demonstrate  the  correct  way  of  isolating  a  malfunction;  this 
is  done  automatically  in  some  simulators. 

The  manufacturers  of  2-D  simulators  have  developed  soft¬ 
ware  packages  and  computer  and  support  equipment  that  can  be 
used  with  a  number  of  different  panels.  This  has  led  us  to 
distinguish  between  what  later  in  discussing  costs  we  call 
"standard"  and  "non-standard"  maintenance  simulator  systems. 
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Standard  systems,  whether  they  are  2-D  or  3-D  simulators,  are 
likely  to  cost  less  than  non-standard  systems. 

A  3-D  maintenance  simulator  looks  and  performs  very  much 
like  the  actual  equipment  it  mimics.  If  it  is  a  test  bench, 
it  will  be  possible  to  connect  components  for  calibration, 
checkout,  and  tests  needed  to  identify  malfunctions.  It  will 
differ  from  actual  equipment  in  that  it  will  be  ruggedized  to 
withstand  student  abuse  and  to  prevent  exposing  students  to 
dangerous  electrical  currents  or  hydraulic  pressures.  The 
simulator  may  not  contain  all  the  components  present  in  the 
actual  equipment,  particularly  those  that  are  not  relevant  to 
its  maintenance;  if  the  equipment  contains  many  identical  com¬ 
ponents,  only  some  will  be  represented.  These  components  may 
be  precise  physical  copies;  in  some  cases,  they  are  only  accu¬ 
rate  photographs  (etched  on  plastic  or  metal)  with  active  test- 
points  for  making  test  measurements.  Being  under  computer  con¬ 
trol,  all  components  perform  or  respond  as  if  they  were  actual 
equipment;  components  may  be  tested,  removed,  and  replaced. 

A  3-D  simulator  permits  "hands  on"  practice  in  the  manual 
maintenance  skills  not  possible  on  most  2-D  simulators;  it 
also  has  greater  physical  similarity  to  the  actual  equipment. 
Whether  or  noc  greater  physical  similarity  increases  the 
effectiveness  of  training  is  not  considered  in  this  discussion. 

E.  OTHEP  INFLUENCES  ON  MAINTENANCE 

Many  factors  beyond  training  and  the  use  of  actual  or  simu¬ 
lated  equipment  can  profoundly  influence  our  ability  to  maintain 
military  equipment.  These  are  noted  here  but  they  extend  far 
beyond  the  scope  of  this  paper.  Among  these  factors  are  the 
quality  of  personnel  recruited  by  the  military  Services  (and 
thereby  available  for  training  as  maintenance  technicians), 
policies  used  by  the  Services  to  assign  recruits  to  various 
occupational  specialties  (thereby  influencing  the  quality  of 
personnel  who  become  maintenance  technicians),  the  amount  and 
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type  of  training  to  be  accomplished  at  technical  schools  (as 
distinct  from  that  to  be  accomplished  on  the  job),  and  the 
complexity  of  the  information  that  must  be  acquired  in  order 
to  accomplish  maintenance.  Some  factors  that  influence  mainte¬ 
nance  have  little  to  do  with  personnel  and  training;  these 
include  equipment  design  and  maintenance  policy.  The  design 
of  equipment  influences  both  the  need  for  maintenance  (mean 
time  between  failure),  and  the  means  for  accomplishing  it 
whenever  required  (e.g.,  ease  of  access  to  components,  built- 
in  test  points,  manual  or  automatic  fault  detection).  Maint¬ 
enance  policy  determines  whether  failed  components  should  be 
repaired  or  replaced,  the  availability  of  spare  parts,  tools, 
test  equipment  and  up-to-date  technical  documentation. 

F.  ADVANTAGES  OF  MAINTENANCE  SIMULATORS 

The  advantages  of  simulators  for  training  maintenance 
personnel  have  been  argued  for  more  than  25  years  (e.g.,  R.B. 
Miller  1954,  Gagne  1962,  Lumsdaine  1960,  Valverde  1968,  Kinkade 
and  Wheaton  1972,  G.G.  Miller  1974,  Montemerlo  1977,  and  Fink 
and  Shriver  1978).  The  major  advantage  of  a  maintenance  simu¬ 
lator  is  that,  as  a  training  device,  it  can  be  designed  to  pro¬ 
vide  facilities  important  for  instructing  students,  in  contrast 
to  actual  equipment  that  is  designed  to  operate  effectively  in 
an  operational  environment. 

Maintenance  simulators  can  be  designed  to  include  a  large 
variety  of  faults  with  which  maintenance  personnel  should  be 
familiar,  including  faults  that  cannot  be  demonstrated  con¬ 
veniently  on  actual  equipment  trainers  or  that  occur  rarely 
in  real  life.  All  modern  maintenance  simulators  incorporate 
some  type  of  computer  support.  Thus,  the  symptoms  of  many 
types  of  complex  faults  can  be  stored  in  the  computer  and  se¬ 
lected  simply  by  a  control  setting  on  the  instructor's  console. 
Computer-supported  equipment  can  also  record  what  the  student 
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does,  thereby  reducing  the  need  for  constant  observation  by  the 
instructor.  The  instructor  can  use  information  collected  by 
the  computer  to  guide  each  student;  a  computer  can  also  assist 
the  student  without  an  instructor's  intervention.  Records  of 
student  performance  and  achievement  can  be  maintained  automati¬ 
cally.  Simulators  can  be  made  rugged  enough  to  sustain  damage 
or  abuse  by  students  and  thus  provide  greater  reliability  and 
availability  in  the  classroom  than  is  often  possible  with  actual 
equipment.  Training  which  would  be  avoided  because  of  safety 
reasons,  e.g.,  exposure  cf  students  to  dangerous  electrical 
charges  or  hydraulic  pressures,  can  be  undertaken  with  little 
risk  with  a  simulator.  If  students  using  such  equipment  com¬ 
plete  their  training  in  less  time,  as  has  often  been  the  case 
with  computer-based  methods  of  instruction,  there  are  potential 
c'ist  benefits  due  to  savings  in  student  time,  increased  student 
throughput,  and  reduced  need  for  instructors  and  support  per¬ 
sonnel. 

As  noted  above,  a  simulator  need  not  contain  all  the  com¬ 
ponents  found  in  the  actual  equipment.  Thus,  it  is  often  pos¬ 
sible  to  build  a  simulator  that  offers  greater  flexibility  and 
capacity  for  training  at  a  cost  less  than  that  for  an  actual 
equipment  trainer. 

G.  DISADVANTAGES  OF  MAINTENANCE  SIMULATORS 

There  are  also  some  disadvantages  to  the  use  of  simulators. 
The  procurement  of  maintenance  simulators  necessarily  involves 
costs  to  design  and  build  this  special  equipment,  and  to  develop 
course  materials,  maintenance  procedures,  and  documentation. 

The  types  of  training  provided  by  simulators  may  not  provide 
the  student  with  all  the  skills  needed  to  maintain  operational 
equipment,  an  outcome  that  seems  assured  when  actual  equipment 
is  used  for  training.  A  simulator  may  not  be  ready  when  needed 
for  training  because  its  design  and  development  requires  some 


effort  in  addition  to  or  at  least  parallel  to  that  needed  for 
the  actual  equipment  (which  is  being  produced  as  part  of  some 
system);  modifications  in  the  design  of  the  actual  equipment 
may  delay  completion  of  the  simulator,  if  it  also  must  be  modi¬ 
fied.  If  there  are  many  and  frequent  modifications,  the  orig¬ 
inal  simulator  may  not  resemble  the  operational  equipment 
closely  enough  to  be  useful  for  training. 

Data  on  the  effectiveness  and  cost  of  maintenance  simula¬ 
tors  and  actual  equipment  trainers  are  considered  in  the  follow¬ 
ing  chapters. 


THE  EFFECTIVENESS  OF  MAINTENANCE  SIMULATORS 


1 1 . 

The  purpose  of  maintenance  training,  whether  with  simulators 
or  actual  equipment,  is  to  qualify  technicians  to  maintain  equip¬ 
ment  in  the  field.  In  fact,  however,  the  effectiveness  of  main¬ 
tenance  simulators  for  training  technicians  has  fce.m  compared 
to  that  of  actual  equipment  only  on  the  basis  of  student  perform¬ 
ance  at  school  and  not  on  the  job;  there  is  one  exception  to  this 
general  statement  ( Cicch ine  1 1  i  ,  Harmon,  Keller,  et  al.,  1980). 

The  lark  of  job  performance  data  to  validate  training  (and 
other  activities  relevant  to  personnel,  such  as  recruitment, 
selection,  and  reimbursement)  applies  generally  to  all  types  of 
military  traininq  and  not  only  maintenance  training. 

A.  EFFECTIVENESS  OF  MAINTENANCE  SIMULATORS  AT  SCHOOLS 

We  found  IS  studies,  conducted  over  the  period  of  1967  to 
1980,  that  compare  the  effectiveness  of  maintenance  simulators 
and  actual  equipment  trainers  for  traininq  in  a  variety  of 
courses  at  military  traininq  schools;  these  are  described  in 
nppendix  A.  Only  12  of  these  studies  provide  enouqh  detailed 
information  to  permit  meaningful  comparisons;  these  are  sum¬ 
marized  in  Table  4. 

Relatively  complete  data  were  found  on  five  maintenance 
simulators  evaluated  in  14  different  courses,  e.g.,  radar,  pro¬ 
pellers,  engines,  flight  controls,  FM  tuner,  test  equipment, 
and  the  Hagen  Automatic  Boiler  Control;  most  are  associated 
with  aviation.  These  courses  varied  in  lenqth  from  3  hours  to 
5  weeks  (median  4.7  days,  N  =  12  courses);  the  number  of  subjects 
'rained  with  simulators  in  these  courses  varied  from  6  to  56 
^median  16,  N  =  14  groups);  a  grand  total  of  267  students  was 


31 


TABLE  4.  SUMMARY  OF  STUDIES  ON  THE  EFFECTIVENESS  OF  MAINTENANCE  SIMULATORS,  1967-1980 


involved  in  all  of  these  studies.  Effectiveness  was  evaluated 
by  comparinq  the  scores  of  students  who  used  simulators  with 
those  of  students  who  used  actual  equipment  trainers  in  end-of- 
course  tests.  There  are  13  comparisons;  in  12  of  these,  students 
trained  with  simulators  achieved  test  scores  the  same  as  or 
better  than  those  trained  with  actual  equipment;  in  one  case, 
scores  were  lower.  The  differences,  though  statistically  sig¬ 
nificant,  have  little  practical  significance. 

Cicchinelli,  Harmon,  Keller,  et  al.,  1980,  compared  super¬ 
visors'  ratings  of  on-the-job  performance  of  technicians  trained 
either  with  a  maintenance  simulator  (the  6883  Test  Station  3-D 
Simulator)  or  actual  equipment  trainer.  Two  field  surveys  pro¬ 
vided  data  on  the  job  performance  of  85  and  56  graduates, 
respectively  (some  twice);  these  comprised  74  and  49  percent, 
respectively,  of  the  students  in  the  original  sample  at  school- 
some  course  oraduates  were  on  the  job  for  periods  of  up  to  32 
weeks.  The  supervisors  did  not  know  how  the  students  had  been 
trained.  Their  ratinqs  showed  no  noticeable  difference  between 
the  performance  of  technicians  trained  with  the  simulator  or 
actual  equipment  trainer.  The  abilities  of  the  technicians 
increased  with  amount  of  time  on  the  job. 

The  automated  and  individualized  method  of  instruction  that 
is  an  inherent  characteristic  of  modern  maintenance  simulators 
should  be  expected  to  save  some  of  the  time  students  need  to 
complete  the  same  course  when  qiven  by  conventional  instruction 
(Orlansky  and  String  1979).  Such  time  saving:,  are  reported  in 
three  of  these  studies  (Parker  and  DePaul*  1967;  Rigney,  Towne , 
King,  et  al.  1978;  and  Swezey  1978);  compared  to  the  use  of 
actual  equipment  trainers,  maintenance  simulators  were  found  in 
these  studies  to  have  saved  22,  50,  and  50  percent,  respectively, 
of  the  time  students  needed  to  complete  the  courses.  Although 
no  explanations  are  offered  for  these  time  savings,  one  could 
surmise  that  they  are  due  to  factors  such  as  the  fact  that 
brighter  students  can  complete  a  self-paced  course  faster  than 
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one  qiven  by  conventional,  group-paced  instruction,  that  main¬ 
tenance  simulators  generally  have  greater  reliability  in  the 
classroom  than  do  actual  equipment  trainers,  and  that  instruc¬ 
tors  need  less  time  to  set  up  training  problems  and/or  to  insert 
malfunctions  in  simulators  than  in  actual  equipment  trainers. 

Based  on  questionnaires  administered  at  the  completion  of 
the  courses,  students  favor  the  use  of  simulators  in  9  of  10 
cases  and  are  neutral  in  one.  Instructors  are  equally  divided 
(about  one-third  in  each  category  of  response)  in  being  favor¬ 
able,  unfavorable,  or  neutral  in  their  attitude  toward  the  use 
of  simulators. 

Overall,  maintenance  simulators  appear  to  be  as  effective 
as  actual  equipment  trainers  for  traininq  military  personnel  at 
schools;  there  is  only  one  contrary  finding.  Some  of  the  pre¬ 
sumed  advantages  of  simulators  were  not  examined  in  these  studies 
and  therefore  cannot  be  evaluated,  e.g.,  their  ability  to  teach 
students  how  to  correct  a  wider  variety  of  malfunctions  than 
can  be  done  with  actual  equipment,  their  superior  availability 
compared  to  actual  equipment  trainers,  and  their  ability  to  meas¬ 
ure  and  report  student  performance  both  to  students  and  instruc¬ 
tors.  The  findings  do  not  suggest  ways  in  which  the  use  of 
maintenance  simulators  could  be  improved  or  where  their  use  is 
likely  to  be  more  effective.  There  are  no  cases,  except  for 
Cicchinelli,  Harmon,  Keller,  et  al.  1980,  where  the  effect  of 
traininq  upon  iob  performance  is  examined;  they  found  no  differ¬ 
ence  between  a  simulator  and  an  actual  equipment  trainer;  how¬ 
ever,  Cicchinelli,  Harmon,  Keller,  et  al.  do  not  report  the 
amount  of  transfer  of  traininq  from  school  to  the  job,  i.e., 
transfer  effectiveness  ratios. 

B.  RELEVANT  DATA  FROM  COMPUTER-BASED  INSTRUCTION 

Modern  maintenance  simulators  can  provide  individualized 
instruction  on  a  series  of  prescribed  lessons.  They  can  also 
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measure  student  performance  and  see  that  the  student  does  not 
qo  to  a  new  lesson  until  he  has  mastered  the  preceding  ones. 

The  instructional  strategies  employed  in  these  simulators  are 
derived  from  widely  used  methods  of  instruction  called  computer- 
assisted  and  computer-managed  instruction;  both  are  individual¬ 
ized  and  self-paced  in  nature  and  use  computers  to  monitor 
student  proqress.  In  computer-assisted  instruction  (CAI),  all 
the  instructional  material  is  stored  in  a  computer  and  presented 
to  the  student  in  a  controlled  manner,  e.g.,  via  a  cathode  ray 
tube  or  a  visual  pro  jection  device  with  random  ac  'ess  to  a  larqe 
reservoir  of  slides.  The  student  responds  to  this  material  by 
touchinq  portions  of  the  screen  sensitive  to  touch  or  by  using 
a  keyboard  or  teletypewriter.  In  computer-managed  instruction 
( CM  I ) ,  the  lessons  are  performed  away  from  the  computer  in  a 
learninq  carrel  or  on  a  laboratory  bench  set-up.  The  niudent 
takes  a  test  at  the  completion  of  each  lesson;  the  answers,  on 
a  sheet,  arc  scored  by  the  computer  which  then  directs  the  stu¬ 
dent  to  a  new  lesson  or  to  additional  practice  on  the  current 
one . 

CAI  and  CMI  systems  are  not  maintenance  simulators  but 
they  have  heen  used  to  provide  certain  aspects  of  maintenance 
traininq,  e.g.,  knowledge  of  operating  principles,  trouble¬ 
shooting  procedures,  fault  identification,  and  the  knowledge 
aspects  of  remove  and  replace  actions  (i.e.,  what  the  technician 
should  do  after  a  fault  is  identified  rather  than  replace 
actual  parts).  Knowledge  about  maintenance  procedures  can  be 
acquired  on  a  CAI  and  CMI  system,  but  this  is  accomplished  with 
less  fidelity  and  with  little  of  the  hands-on  experience  that 
can  be  provided  by  a  maintenance  simulator,  particularly  of 
the  3-D  variety.  Elsewhere  in  this  paper,  where  we  consider 
coats,  we  characterize  some  maintenance  simulators  as  CAI-like. 

In  a  previous  study,  the  authors  examined  the  cost- 
effectiveness  of  computer-based  instruction  in  military  traininq 
(Orlansky  and  String,  1979).  Some  of  the  courses  on  which 
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effectiveness  data  were  available  involved  instruction  similar 
to  that  provided  on  maintenance  simulators,  i.e.,  basic  elec¬ 
tronics,  vehicle  repair,  fire  control  s\  tern  maintenance,  pre¬ 
cision  measuring  equipment,  and  weapons  >chanics .  Data  on 
student  achievement  in  these  courses  are  resented  in  Table  5; 
there  are  28  data  points  which  compare  conventional  instruction 
to  the  use  of  CAI  and  two  to  CMI.  Student-  achievement  in  these 
courses  at  school  with  CAI  or  CMI  was  the  ume  as  or  superior 
to  that  provided  by  conventional  instruction;  the  amount  of 
superior  performance,  when  present,  had  •  > f tie  practical  signi- 
f  icance . 

Data  on  the  amount  of  student  time  saved  b/  CAI  or  CMI  in 
these  courses,  compared  to  conventional  instruction,  are  shown 
in  Table  6;  there  are  30  data  points.  The  amount  of  time  saved 
by  computer-based  instruction  varied  rrom  -32  to  59  percent, 
with  a  median  value  of  28  percent. 

These  data  on  student  achievement  and  on  student  time 
snvinqs  with  computer-based  instruction  are  consistent  with 
that  reported  above  for  maintenance  simulators,  Orlansky  and 
String  (1979)  found  that  students  favor  computer-based  instruc¬ 
tion  while  instructors  do  not.  They  also  found  that  computet- 
based  instruction  may  increase  student  attrition,  a  matter  not 
considered  so  far  in  any  study  of  maintenance  simulation. 

In  summary,  the  data  show  that  maintenance  simulators  are 
as  effective  as  actual  equipment  when  used  for  training  mili¬ 
tary  technicians.  These  results  are  consistent  with  the  results 
of  studies  of  computer-assisted  and  computer-managed  instruction 
in  courses  that  provide  technical  information  similar  to  that 
provided  in  maintenance  training.  A  few  studies  show  that 
maintenance  simulators  save  student  time  but  most  studies  did 
not  address  this  issue.  Students  favor  the  use  of  maintenance 
simulators;  instructors  favor,  are  neutral  about,  or  do  not 
favor  such  simulators  in  about  equal  numbers. 
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TABLE  5.  STUDENT  ACHIEVEMENT  AT  SCHOOL  IN  COURSES  RELEVANT  TO  MAINTENANCE 
CAI  AND  CM  I  COMPARED  TO  CONVENTIONAL  INSTRUCTION 
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Claims  have  been  made  that  maintenance  simulators  are 
superior  to  actual  equipment  for  training  because  of  their 
capability  to  demonstrate  more  malfunctions,  provide  greater 
freedom  from  breakdown  in  the  classroom  environment,  provide 
an  opportunity  to  save  instructor  time,  and  so  on.  No  studies 
were  found  that  examine  these  capabilities.  No  data  were  found 
on  student  attrition  when  simulators  are  used. 

C.  PERFORMANCE  OF  TECHNICIANS  IN  THE  FIELD 

The  effectiveness  of  maintenance  training  is  determined 
ultimately  by  how  well  maintenance  personnel  perform  in  the 
field  rather  than  at  school.  Only  Cicchinelli,  Harmon,  Keller, 
et  al.  1980,  among  the  studies  we  were  able  to  find,  compared 
the  performance  of  students  trained  with  a  simulator  (the  6883 
Test  Station  3-D  Simulator)  or  actual  equipment  trair er  for 
varying  periods  of  time  after  leaving  school.  According  to 
ratings  made  by  supervisors,  no  differences  were  found  between 
both  groups  of  students. 

The  military  services  use  five  large  data  management 
systems  to  provide  detailed  information  on  the  current  main¬ 
tenance  status  of  military  equipment.  These  data  systems 
are  identified  below: 

Service  _  _  Maintenance  Management  System _ 

Army 

Navy 

Navy 

Air  Force 


TAMMS 

Ships'  3-M 
Aviation  3-M 

66-1  and  66-5 


The  Army  Maintenance  Management 
System 

The  Naval  Ships'  Maintenance  and 
Material  Management  System 

Naval  Aviation  Maintenance  and 
Material  Management  System 

Air  Force  Maintenance  Management 
Systems 
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We  examined  the  possibility  of  using  data  available  in  these 
systems  to  describe  the  performance  of  maintenance  technicians 
in  the  field  (see  String  and  Orlansky,  1981).  If  this  yielded 
useful  information,  we  might  be  able  to  compare,  for  example, 
the  real  short-  and  long-term  effects  of  training  personnel 
with  maintenance  simulators  or  with  actual  equipment  trainers. 

We  know,  as  was  shown  earlier  in  the  chapter,  that  both  are 
about  equally  effective  at  school. 

As  presently  constituted,  these  systems  cannot  provide 
information  useful  for  assessing  the  effectiveness  of  alter¬ 
native  methods  of  training.  In  a  more  general  sense,  this 
applies  also  to  information  needed  to  validate  many  personnel 
practices,  such  as  recruiting,  selection,  and  policy  on  pay  and 
allowances.  The  names  of  individuals  who  performed  maintenance 
actions  are  not  kept  in  the  records  maintained  in  the  central 
data  files.  The  ability  to  identify  and  track  individuals  is 
a  mandatory  requirement  in  any  attempt  to  relate  method  of 
training  with  subsequent  performance.  This  type  of  data  is 
kept  only  at  the  field  activities  but  it  is  discarded  after 
6  months.  The  use  of  maintenance  records  with  personal  iden¬ 
tification  for  analytical  purposes  would  require  special  methods 
of  processing  in  order  not  to  infringe  on  provisions  of  the 
Privacy  Act.  Even  so,  such  records  are  not  precise  enough  to 
distinguish  what  parts  of  a  maintenance  action  were  performed 
by  a  particular  individual,  particularly  when  the  work  is 
performed  over  more  than  one  shift.  The  practice  of  cross-skill 
maintenance,  to  train  individuals  to  maintain  a  wide  variety 
of  equipment  under  combat  conditions,  assigns  individuals  to 
tasks  for  which  they  were  not  trained  at  school  and  it  would 
complicate  any  analytical  effort.  In  brief,  it  was  concluded 
that  presently  available  maintenance  data  records  can  not  be 
used  to  assess  the  effectiveness  on  the  job  of  various  methods 
of  training  at  school;  it  is  conceivable  that  these  systems 
could  be  modified  t.o  provide  the  data  that  would  be  needed. 
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That  the  performance  of  maintenance  technicians  affects 
the  quality  of  maintenance  can  hardly  be  doubted.  A  few  studies 
have  examined  this  possibility  by  analyzing  selected  data  on 
components  removed  for  replacement  or  repair  that  were  found 
later  not  to  contain  any  malfunction.  These  studies  examine 
data  produced  by  a  group  of  technicians  in  a  particular  work 
center;  they  do  not  review  the  performance  of  individual  techni¬ 
cians  and  do  not  address  the  method(s)  by  which  these  technicians 
were  trained. 

Findings  from  seven  studies  are  summarized  in  Table  7. 

All  involve  corrective  maintenance  at  the  organizational  level, 
although  one  also  involved  intermediate  maintenance.  Most 
concern  maintenance  of  aircraft,  a  few  of  surface  vehicles 
The  periods  of  observation  are  relatively  long  (6  months  or  1 
year;  one  is  for  only  one  month).  The  removal  or  non-faulty 
parts,  in  these  studies,  accounted  for  4  to  43  percent  of  all 
corrective  maintenance  actions  and  9  to  32  percent  of  all 
maintenance  man-hours.  One  study  (Gold,  Kleine,  Fuchs,  et  al., 
1980)  found  instances  where  faulty  parts  were  not  removed  and 
where  good  parts  were  damaged  during  corrective  maintenance. 

These  findings  suggest  strongly  that,  properly  modified,  the 
maintenance  data  systems  might  provide  data  on  human  performance 
useful  for  validating  different  methods  of  training.  Even  so, 
it  is  well  to  recognize  that  not  all  instances  of  removal  of 
good  parts  necessarily  imply  inadequate  performance  of  tech¬ 
nicians.  Such  removals  could  also  be  due  to  inadequate  test 
equipment  that  cannot  distinguish  between  good  and  bad  parts. 

It  is  also  possible  that,  when  under  great  pressure  to  return 
equipment  to  an  operational  status,  technicians  may  deliberately 
remove  and  replace  a  large  number  of  components  just  to  make 
sure  that  the  faulty  ones  have  been  eliminated.  Validation 
of  training  devices  and  procedures  would  probably  need  more 
data  on  job  performance  than  just  that  concerning  the  unneces¬ 
sary  removal  of  good  parts. 
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TABLE  7.  SUMMARY  OF  STUDIES  OF  ORGANIZATIONAL  ECHELON  CORRECTIVE  MAINTENANCE 

WHERE  NON-FAULTY  PARTS  WERE  REMOVED 


records,  > *.-'!>  r.n*  tc  corpare  defects  reported 


III. 


COSTS  OF  MAINTENANCE  SIMULATORS 


> 


A.  INTRODUCTION 

This  chapter  discusses  the  costs  of  maintenance  training 
simulators.  Three  classes  of  simulators  are  defined,  their 
characteristics  and  uses  within  the  Services  are  discussed,  and 
a  structure  of  data  for  analyses  of  their  costs  is  formulated. 
Available  cost  data  tor  maintenance  of  simulators  are  discussed 
in  terms  of  the  problems,  with  respect  to  costs,  that  arise 
from  their  physical  characteristics,  procurement  quantities, 
and  contracting  practices.  The  costs  and  characteristics  of 
selected  simulator  programs  are  presented  in  Appendix  B. 

B.  CLASSES  OF  SIMULATORS 

With  respect  to  the  costs  or  maintenance  training  simula¬ 
tors,  it  is  useful  to  distinguish  among  three  classes  of  de¬ 
vices,  denoted  here  as  "standard"  systems,  "non-standard"  sys¬ 
tems,  and  "CAI-like"  systems.  Differences  among  these  three 
types  lie  in  the  following  areas: 

•  Physical  characteristics, 

•  Complexity  and  cost, 

•  Extent  of  use  within  the  Services  (i.e.,  the  inventories 
of  devices  in  use  and  under  contract),  and 

•  Contracting  practices  employed  in  their  procurement  (and 
hence  cost  data  that  are  available). 
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1. 


Standard  Sy ste ms 

The  critical  distinction  between  standard  and  ether  classes 
of  maintenance  simulators  is  standardization  of  the  physical 
conf iquration.  Simulators  of  this  class  consist  of  two  elements 
one  element,  called  here  the  "general  simulation  system"  consti¬ 
tutes  a  generalized  and  adaptable  (but  incomplete)  simulation 
capability  that  can  satisfy  a  wide  range  of  specific  training 
applications.  The  second  element,  that  tailors  the  general 
simulation  system  to  a  particular  training  application,  is  typi¬ 
cally  limited  to  courseware  and  pictorial  or  other  representa¬ 
tions  (i.e.,  the  simulation  model)  of  the  particular  equipment 
being  simulated.  Standard  systems  were  the  earliest  type  to  be 
used  for  maintenance  training  and  are  the  only  class  to  achieve 
extensive  use.  The  three  Services  have  procured  close  to  650 
simulators  for  nearly  200  separate  training  applications  (train¬ 
ing  courses  or  course  segments). 

Only  four  companies  have  manufactured  standard  maintenance 
simulator  systems:  Educational  Computer  Corporation  (ECC); 
Burtek,  Inc.;  Ridgeway  Electronics,  Inc.;  and  Lockheed  Aircraft 
Services  Co.  (LAS).  For  all  but  Ridgeway,  this  type  of  simula¬ 
tor  is  only  one  of  several  product  lines;  and  for  all  but  LAS, 
these  companies  manufacture  only  educational  and  traininq  equip¬ 
ments. 

Compared  with  the  other  classes  of  simulators,  the  stand¬ 
ard  systems  are  generally  low  in  cost  and  limited  in  terms  of 
the  complexity  of  processes  that  can  be  simulated.  Development 
of  particular  traininq  applications  typically  involves  sma 1 1 
risks.  With  few  exceptions,  these  devices  have  been  procured 
throuqh  fixed-price  contracts. 

2 .  Cost  Impact  of  Standardization 

The  four  manufacturers  have  produced  six  standard  simu¬ 
lator  systems  or  models.  The  elements  that  are  typically  com¬ 
mon  to  a  model  consist  of  data-process i ng  hardware  (a  central 


processor  and  a  partial  set  of  input/output  devices) ,. -the  soft¬ 
ware  operating  system,  audio-visual  devices,  and  structures  for 
housinq  all  the  components  of  the  simulator.  Taken  together, 
these  are  generally  referred  to  as  the  "mainframe"  or  "console" 
The  components  that  are  tailored  to  the  particular  application 
consist  of  other  input/output  devices  (typically  a  display 
panel  depicting  the  operational  and  test  equipment  being  simu¬ 
lated)  and  courseware  in  the  form  of  an  application  program 
(contained  in  magnetic  tape,  disc,  or  plug-in  proqrammable-read 
only-memory  (PROM)  units] . 

The  size  and  structure  of  display  media  may  vary  within  a 
sinqle  model,  and  the  same  simulation  application  may  be  pro¬ 
duced  with  two  sizes  of  display  panels--one  for  classroom 
demonstrations  and  one  for  individual  use.  Advances  .in  micro¬ 
processor  technology  appear  to  have  fostered  further  variations 
within  a  model  while  retaining  the  essential  attributes  of 
standardization.  One  ECC  model  has  been  delivered  with  pro¬ 
cessor  memories  ranging  between  16  and  48  thousand  bytes.  The 
variation  in  memory  size  has  permitted  corresponding  variations 
in  complexity  of  simulation  and  the  use  of  audio/visual  devices 
For  example,  the  4 8-thousand-by te  devices  procured  through  an 
Army  contract  for  XM- 1  tank  training  will  drive  a  cathode  ray 
tube  (CRT),  printer,  random-access  slide  projector,  and  an 
audio  device  in  addition  to  the  normal  si  mu la tor  display 
panel.  A  contemporary  Navy  contract  (for  entry-level  skill 
training)  specifies  the  same  model  with  a  16 -thousand-byte 
memory  and  with  only  the  display  pane). 

The  physical  arrangement  of  standard  systems  appears 
especially  adaptable  for  2-dimensional  trainers.  However,' 

3 -d imens ional  simulation  is  possible. 

One  impact  of  standard i zat ion  is  interchangeability,  and 
this  serves  to  reduce  costs  of  both  manufacture  and  repair. 
Individual  consoles  may  be  easily  modified  to  different  train¬ 
ing  applications.  Two  of  the  standardized  models  are  designed 
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so  that  the  tailored,  specific  components  of  any  training  4 

application  (the  display  panel  and  courseware)  can  be  mated 
with  a  single  console  in  the  classroom  or  laboratory;  a  third 
model  can  be  ordered  with  either  classroom-changeable  or  fixed 
panels.  As  a  result,  a  single  console  may  be  used  in  a  number  4 

of  training  applications  at  the  same  location.  One  of  these 
models  (the  ECU  provides  the  bulk  of  the  simulators  used  in 
Naval  aviation  weapon-specific  training.  Naval  aviation  train¬ 
ing  is  organized  so  that  both  maintenance  and  a  part  of  pilot  4 

ground  training  for  one  model  of  aircraft  are  conducted  at  the 
same  Naval  Air  Station  (NAS)  by  Naval  Air  Maintenance  Training 
Detachments  (NAMTD) .  While  a  large  number  of  different  dis¬ 
play  panels  (up  to  25)  may  be  employed  for  maintenance  training  4 

by  one  detachment,  a  NAMTD  will  generally  have  no  more  than  two 
main  frames  that  will  be  shared  by  all  pilot  and  maintenance 
training  courses. 

The  more  important  cost  impact  of  standardization  lies  in  4 

the  commonality  of  system  software.  Available  evidence  points 
to  the  programming  and  programming  design  effort  as  the  major 
cost  of  non-standard  simulator  devo lopment .  This  high  cost 

provides  a  strong  incentive  for  producers  to  develop  a  single  I 

basic  software  system  that  is  both  comprehensive  and  adaptable 
to  a  w'de  range  of  potential  training  applications.  Develop¬ 
ment  of  such  a  software  system  reduces  the  programming  asso¬ 
ciated  with  a  particular  training  application  to  a  relatively  j 

small  set  cf  courseware  written  in  a  high-level  and  relatively 
simple  language  that  may  (in  the  case  of  maintenance  simula¬ 
tion)  reduce  to  a  sequential  coding  of  the  maintenance  proce¬ 
dures/steps  set  out  in  technical  orders.  I 

Commonality  of  software  is  the  distinguishing  aspect  of 
standardization ,  and  manufacturers  have  placed  a  heavy  emphasis 
on  developing  versatile  software  packages.  Once  developed,  they 
are  tightiv  held,  considered  proprietary,  and  may  (at  least  in  j 

part)  be  hardwired  into  simulators.  In  addition,  the  software 
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packages  have  been  retained  while  other  features  of  the  general 
simulation  systems  have  been  allowed  to  change.  ECC  has  pro¬ 
duced  two  generations  of  standard  training  simulators;  the 
later  one  employs  an  advanced,  higher  capability  processor  and 
has  a  quite  different  physical  appearance,  but  uses  the  earlier 
software  package.  LAS  produces  two  models  of  simulators  that 
employ  different  types  of  display  panels.  Yet,  the  two  employ 
the  same  software  system  and  seem  best  considered  as  a  single 
generalized  system. 

3 .  Non-Standard  Systems 

The  non-standard  systems  present  a  picture  that  is  quite 
different  from  the  standard  systems.  Seventeen  non-standard 
programs  (discussed  in  Section  C,  below)  have  been  initiated; 
with  one  exception,  each  appears  to  involve  a  complete  (i.e., 
ground  up)  development  effort.  Taken  as  a  group,  their  out- 
standinq  characteristic  is  diversity,  encompassing  different 
contractors  and  types  of  contracts,  proaram  purpose,  numbers 
of  devices  manufactured,  physical  characteristics,  complexity, 
and  cost. 

Two  programs  (the  AT  Trainer  and  AN/TSP-43E  radar)  have 
been  in-house  projects  at  military  installations  while  the 
remaining  15  have  been  contracted  to  one  or  more  firms.  The 
15  contracteo  programs  have  involved  10  firms  as  principal  con¬ 
tractor;  one  company  (Honeywell)  has  played  this  role  on  five 
projects.  Only  one  firm  (ECC)  has  also  had  experience  in  pro¬ 
ducing  a  standard  device;  four  firms  (Grumman,  RCA,  Hughes,  and 
Sperry)  also  produced  the  tactical  equipment  being  simulated. 

Three  programs  (the  MA-3,  A- 7  HUD,  and  6883  Test  Benches) 
have  research  in  maintenance  simulation  as  their  principal  pur¬ 
pose  and  employ  cost-plus- f ixed- fee  (CPFF)  contracts.  The  other 
14  programs  (including  the  two  in-house  programs)  serve  main¬ 
line  training.  Ten  of  the  12  that  were  contracted  were  funded 
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through  either  fixed-price-incentive-fee  (FPIF)  or  firm-fixed- 
price  (FFP)  contracts;  the  remaining  two  (MK  92  FCS  and  Fire- 
finder)  employed  cost-plus  contracting. 

When  completed  (as  currently  planned),  the  17  programs 
will  result  in  the  development  of  47  unique  simulations  and 
delivery  of  687  units,  i.e.,  trainers.  The  Mk  92  FCS,  CIWS, 
and  F-16  programs  will  be  responsible  for  30  of  the  different 
simulations  and  632  of  the  trainers;  both  the  CIWS  and  F-16 
address  training  in  a  number  of  skills  for  a  single  weapon  sys¬ 
tem  and  will  result  in  the  development  of  a  family  of  devices 
with  extensive  commonality,  'ather  similar  to  the  standard 
systems.  Typically,  the  other  programs  are  concerned  with 
single  training  applications  and  a  single  training  device. 

The  physical  characteristics  of  the  non-standard  simula¬ 
tors  appear  to  be  similarly  diverse.  There  are  two-  and  three- 
dimensional  trainers.  Since  software  is  normally  closely 
held  by  contractors,  wide  variability  car.  be  expected.  Fur¬ 
ther,  since  a  non-standard  system  typically  simulates  only 
one  tactical  system,  it  is  not  necessary  to  provide  a  defini¬ 
tive  separation  between  software  and  courseware  functions. 

The  total  program  costs  of  the  non-standard  systems 
(adjusted  to  current  price  levels)  differ  by  factors  of  up 
to  300:1,  and  the  average  costs  of  devices  differ  by  factors 
of  up  to  40:1. 

4 .  CAI-Likc  Systems 

A  CAl-like  maintenance  simulator  is  a  computer-assisted  in¬ 
struction  (CAI)  system  with  courseware  designed  specifically  to 
train  maintenance  skills.  A  typical  CAI  system  uses  a  2-dimen¬ 
sional  display  (CRT  and/or  random  access  slide  or  microfiche 
projector)  to  present  lesson  materials  (pictures  of  equipment 
and  the  like)  under  control  of  a  computer  that  also  monitors 
student  proqress,  prescribes  lessons,  and  scores  tests.  When 
adapted  to  maintenance  training,  the  CAI  features  are  retained, 
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and  the  trainer  may  also  employ  3-dimensional  depictions  of 
equipment . 

One  experimental  system  of  this  class  (the  Rigney  Trainer) 
has  been  built,  and  two  other  systems  have  recently  been  placed 
under  contract.  A  contract  for  the  design  and  fabrication  of 
prototype  units  of  the  Electronic  Equipment  Maintenance  Trainer 
or  (EEMT)  was  awarded  to  Cubic  Corporation  by  the  Navy  Personnel 
Research  and  Development  Center  (NPRDC)  in  August  of  1980,  and 
a  preliminary  design  has  been  formulated.  The  Army  has  let 
several  contracts  for  the  study  of  design  concepts  tor  the  Army 
Maintenance  Training  and  Evaluation  Simulation  System  (AMTESS) 
and  let  a  contract  to  Grumman  Aerospace  in  December  1980  for 
construction  of  two  "breadboard"  units  for  further  evaluation. 

EEMT  is  intended  for  initial  skill  ("A-school")  training, 
primarily  in  electronics.  It  is  to  provide  both  2-dimensional 
displays  (generated  through  a  cathode  ray  tube)  and  3-dimen¬ 
sional  simulations  and  is  to  be  capable  of  simulating  a  variety 
of  particular  electronic  systems.  This  latter  capability  is  the 
basis  for  distinguishing  CAI-like  from  the  other  classes  of  main' 
tenance  simulators.  The  software  system  must  be  comprehensive 
and  adaptable  (as  in  the  case  of  standard  systems).  In  addi¬ 
tion,  both  the  software  system  and  the  courseware  must  be  more 
extensive  since  they  must  also  provide  the  information  that 
would  be  contained  on  the  display  panels  of  simulators  that  are 
tailored  to  a  particular  training  application  (whether  standard 
or  nonstandard  types) . 

The  only  information  available  to  this  project  on  the  costs 
of  CAI-like  systems  is  contained  in  the  cost  proposal  for  the 
EEMT  system.  In  this  proposal,  requirements  for  labor  (of  all 
types)  were  stated  in  terms  of  hours,  with  insufficient  infor¬ 
mation  to  convert  them  to  dollar  costs  to  develop  an  estimate 
of  total  program  costs.  As  a  result,  the  CAI-like  systems  are 
not  discussed  further  in  this  chapter. 
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SERVICE  USE  OF  MAINTENANCE  SIMULATORS 


C  . 


Service  inventories  of  maintenance  simulators  show  quite 
diverse  policies  reqardinq  their  use.  Table  8  presents  a  sum¬ 
mary  oE  the  diEEerent  types  and  total  quantities  oE  trainers 
procured  by  each  Service  and  distinquishes  between  Naval/Marine 
Corps  aviation  and  other  Navy  and  Marine  Corps  usage.  The 
diEEerences  in  reliance  on  simulation  are  more  evident  in  the 
case  oE  standard  systems.  The  bulk  oE  Navy  aEloat  and  Air  Force 
non-standard  systems  result  Erom  the  Mk-92,  CIWS  (Phalanx), 
and  F-16  proqrams.  In  the  absence  oE  these  two  programs,  there 
would  be  little  diEEerence  amonq  any  oE  the  Services.  It  is 
noteworthy,  thouqh,  that  these  large  proqrams  are  in  areas  that 
have  shown  the  least  use  in  the  standard  systems  in  the  past. 

TABLE  8.  SUMMARY  OF  MAINTENANCE  TRAINING  SIMULATORS  DELIVERED 
AND  ON-CONTRACT  SINCE  1972,  BY  SERVICE  AND  BRANCH 


Service  and  Branch 

Standard  Systems 

Non-Standard  Systems 

Number  of 
Different 
Devices 

Total 
Number 
of  Units 

Number  of 
Different 
Devices 

Total 
Number 
of  Units 

Navy/Marine  Corps 

137 

354 

6 

11 

Aviation 

Marine  Corps  Ground 

27 

129 

0 

0 

Forces 

Navy  Afloat 

4 

10 

25 

581 

Army 

24 

158 

2 

34 

Air  Force 

_ 2 _ 

2 

14 

61 

Total s 

194 

653 

47 

— 

687 

One  result  oE  standardization  is  that  it  is  diEEicult  to 
identity  speciEic  simulator  development  or  procurement  proqrams. 
The  standard  devices  that  provide  training  Eor  a  particular  sys¬ 
tem  (e.g.,  a  given  model  oE  aircraft)  may  have  been  procured 
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through  several  contracts  initiated  at  different  times.  A 
single  contract  may  encompass  varying  quantities  of  devices  for 
several  equipments  and  include  procurement  of  the  general  simu¬ 
lation  system  as  well  as  the  unique  components  for  different 
training  applications.  As  a  result,  the  discussion  of  usage  of 
these  devices  is  limited  to  procurement  quantities  by  Service 
and  according  to  skill  areas  trained  and  to  the  market  shares 
of  four  contractors.  The  non-standard  systems  are  developed 
and  procured  within  well-defined  programs  that  are  related  to 
particular  simulator  systems  and  training  applications  and  are 
discussed  in  that  context. 

1 .  Standard  Systems 

The  first  procurements  of  standard  maintenance  simulators 
occurred  in  the  1972-1973  period  when  limited  quantities  were 
delivered  to  the  Air  Force,  to  the  Navy  for  surface  traininq, 
and  to  the  Marines  for  ground  forces  training.  The  first 
deliveries  for  Naval/Marine  Corps  aviation  traininq  occurred 
a  few  years  later,  and  since  that  time  this  training  has  become 
the  most  extensive  user  of  standardized  systems.  The  current 
inventory  of  354  devices  accounts  for  70  percent  of  the  dif¬ 
ferent  simulations  and  nearly  55  percent  of  the  total  units 
employed  in  military  training. 

The  Marine  Corps  was  the  earliest  service  to  contract  for 
a  significant  number  of  standardized  systems.  A  1972  contract 
called  for  delivery  of  15  units  (encompassing  11  different 
simulations)  for  training  of  ground  equipment  maintenance; 
this  was  followed  in  1975  with  a  contract  for  114  units  of  27 
different  simulations  (including  reprocurement  of  the  11  types 
of  simulations  contracted  for  in  1972).  The  last  of  the  Marine 
Corps  inventory  was  delivered  in  1976  and  none  have  been  con¬ 
tracted  for  since  that  time. 

The  first  known  Army  use  of  standard  systems  for  mainte¬ 
nance  training  was  in  1977,  with  the  delivery  of  two  devices 
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for  traininq  of  the  Mohawk  propeller  control  system.  Since 
that  time,  the  Army  has  procured  devices  for  training  of  other 
aircraft  systems  and  several  armored  vehicles.  Note  that  the 
numbers  shown  in  Table  8  include  only  devices  that  were  procured 
through  the  Proqram  Manaqer  for  Training  Devices  (PMTRAOE).  The 
devices  listed  in  Table  8  may  be  an  incomplete  listing  as  Army 
management  procedures  allow  traininq  devices  to  be  contracted 
for  throuqh  weapon  system  proqram  offices  and  individual  base 
commands.  Such  devices  are  not  registered  in  a  central  inven¬ 
tory  record  and  cannot  be  readily  identified. 

Both  the  Air  Force  and  Navy  afloat  have  made  little  use  of 
standardized  simulators.  The  Air  Force  procured  one  device  in 
the  early  1970s  to  evaluate  its  use  in  training  AN/ALQ-126  radar 
maintenance  personnel  (as  part  of  a  research  project).  A  second 
device  simulating  the  6883  Test  Bench  was  delivered  in  1980, 
also  for  evaluation  as  part  of  a  research  program.  The  Navy 
procured  five  copies  of  one  device  for  traininq  in  the  tuning 
of  traveling  wave  tubes  in  1973.  The  next  delivery  of  this 
class  of  simulator  (the  Hagen  Automatic  Boiler  Control  Simula¬ 
tor)  was  in  1978  as  part  of  a  research  program  investigating 
training  strategies  for  equipments  the  maintenance  of  which 
requires  personnel  trained  in  different  skill  areas.  The  only 
current  use  of  standard  simulators  for  main-line  traininq  con¬ 
sists  of  two  device.;  tci.  traininq  maintenance  of  the  Trident 
submarine  air-conditioning  and  air-compressor  systems. 

The  standard  systems  have  been  used  for  training  in  a 
variety  of  skill  areas,  as  shown  in  Table  9,  with  traininq 
applications  spread  rather  equally  among  the  broad  groupings  of 
electrical  and  electronic,  propulsion,  and  combinations  of 
mechanical/hydraulic/pneumatic  areas.  This  stands  in  contrast 
with  the  non-standard  systems  discussed  below  where,  excepting 
the  two  large  weapon-system-oriented  programs  (F-16  and  CIWS), 
all  but  the  MA-3  and  DD  963  boiler  programs  have  been  limited 
to  simulation  of  electronic  systems. 
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TABLE  9.  STANDARD  SYSTEMS  DELIVERED  AND  UNDER  CONTRACT, 
BY  SERVICE  AND  SKILL  AREA 


Number  of 

Skill  Area  by  Military  Service 

Di fferent 
Simulation 

■1 

Number 
of  Devices 

Marine  Corps  Ground  Forces 

Electrical 

12 

38 

Propulsion 

1  1 

75 

Hy d ra u 1 i c- Pneuma  t  i  c 

4 

16 

Naval  and  Marine  Corps  Aviation 

Genera  1  Skill  Training3 

Electrical-El ec  tron i c 

10 

78 

Propul sion 

16 

00 

E  1  ec  t ro- Hy dra  ul  i  c 

14 

76 

Weapon-Specific  Training 

Electrical 

18 

24 

El ectroni  c 

23 

34 

Propul S i on 

14 

14 

E 1 ec  tro-Hydrau  1  i  c 

1 

1 

Meehan ical-Hydraul  ic 

29 

35 

Unknown 

12 

12 

Navy  Afloat 

Hydra  u 1 i C - Pneuma  t i c 

2 

2 

Combination  of  Skill  Areas 

1 

3 

Electronic 

1 

5 

Air  Force 

Electronic 

2 

2 

Army 

Aviation 

Electric,!  1 

4 

22 

Electronic 

4 

48 

H  y  d  r  a  u  1  i  c 

1 

2 

Mechanical -Hydraul  ic 

1 

7 

El ectro-Mechanical 

2 

1  4 

Ground 

Electrical-Electronic 

2 

4 

Propulsion 

4 

16 

Hydra  u  l i c 

2 

9 

El ectro-Hydraul i c 

4 

36 

includes  training  in  aircraft  and  ground 

support  equipment. 

A  relevant  point  to  be  seen  in  Table  9  is  that  while  these 
systems  have  found  a  wider  range  of  applications  (in  terms  of 
different  simulations)  for  weapon-specific  Naval  aviation  train¬ 
ing,  one  or  two  units  (trainers)  of  a  given  simulation  will 
satisfy  a  training  requirement.  That  is,  97  different  simula¬ 
tions  (training  applications)  are  satisfied  by  120  devices,  an 
average  of  only  one  and  one-quarter  units  of  each  simulation. 
This  contrasts  with  the  larger  numbers  of  identical  units  re¬ 
quired  for  general  aviation  skill  training  (or  for  training  for 
widely  held  equipments  such  as  those  employed  by  the  Army  and 
Marine  Corps  ground  forces).  Non-recurring  costs  involved  in 
bringing  a  simulation  on-line  are  hiqh  compared  with  the  costs 
of  fabricating  additional  units  of  an  already  designed  simula¬ 
tion,  and  this  relation  has  a  large  impact  on  the  average  costs 
of  simulation  in  training  for  different  types  of  equipments. 

Table  10  shows  the  number  of  standard  systems  delivered 
and  under  contract,  according  to  manufacturer.  ECC  appears  to 
dominate  the  market,  but  the  extent  of  this  domination  is  de¬ 
creasing.  Ridgeway  is  a  new  company  that  appears  to  be  aggres¬ 
sively  marketing  its  system.  As  of  mid-summer  1980,  all  of 
the  107  Ridgeway  devices  shown  were  under  contract,  but  none 
had  been  delivered.  ECC,  by  contrast,  had  undelivered  orders 
for  100  devices.  When  all  of  these  deliveries  are  completed, 
the  percent  of  devices  in  use  that  are  manufactured  by  ECC 
will  drop  from  90  to  75. 

Standard  maintenance  simulators  are  not  major  products  of 
either  Burtek  or  Lockheed  Aircraft  Services.  Burtek  produces 
a  wide  range  of  training  devices  (from  aircraft  evacuation  and 
ejection  seat  trainers  to  automated  study  carrels)  for  both  the 
civilian  and  military  markets.  Lockheed  Aircraft  Service  pro¬ 
vides  a  wide  ranqe  of  aircraft-related  products  and  services 
(including  aircraft  modifications,  full-scale  models,  mock-ups, 
and  training  services)  for  both  military  and  civilian  customers. 
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TABLE  10.  STANDARD  SYSTEMS  DELIVERED  AND  UNDER  CONTRACT, 
BY  MANUFACTURER  SINCE  1 972 


Manufacturer  j 

Educational 

Computer 

Corporation 

Ridgeway 

Electronics, 

Inc. 

Burtek , 

Inc . 

Lockheed 

Aircraft 

Service 

Marine  Corps  Ground  Forces 

Unique  Models 

27 

Total  Held 

129 

- 

-- 

-- 

Naval  &  Marine  Corps  Aviation 

General  Skill  Training 

Unique  Model s 

21 

18 

1 

Total  Held 

119 

103 

12 

-- 

Weapon-Specific  Training 

Unique  Models 

74 

... 

11 

12 

Total  Held 

97 

-- 

11 

12 

Navy  Afloat 

Unique  Models 

i 

3 

Total  Held 

5 

5 

-- 

Army 

Unique  Models 

22 

1 

1 

Total  Held 

152 

4 

2 

- 

USAF 

Unique  Models 

1 

1 

Total  Held 

1 

1 

- 

Total 

Unique  Models 

146 

19 

17 

12 

Total  Held 

503 

107 

31 

12 

2 •  Non-Standard  Systems 

The  non-standard  simulators  are  relatively  recent  develop¬ 
ments.  A  listinq  and  description  of  the  programs  that  have  been 
initiated  to  date  are  shown  in  Tables  11  and  12.  In  fact,  there 
has  been  little  experience  in  training  with  this  class  of  simu¬ 
lator.  Several  of  these  programs  have  yet  to  result  in  deliv¬ 
eries;  for  several  others,  deliveries  have  not  been  completed 
or  deliveries  have  been  of  less  than  complete  or  full-design 
configurations.  There  is  generally  an  installation  and  checkout 
period  and  a  significant  period  between  the  initial  and  final 
deliveries  of  a  program  so  that,  even  where  a  full  configuration 
has  been  delivered,  actual  use  for  training  would  be  less  than 
is  suggested  by  Table  12. 


TABLE  11.  NON-STANDARD  SIMULATOR  PROGRAMS,  DESCRIPTION 


Trainer  Designation 

Description  of  Asso<  lated  Operational  Equipment 

Designation  of 
Operational 
Equipment 

Training  Equipment  Cortractor(s)n 

Naval  Aviation 

VTA'j 

Visual  Target  Acquisition  System  of  F-4N. 

AN/AVP.-H 

Honeywel 1 

AT -1 ra i ner 

All  equipment  maintained  by  AT  rating  on 

E-4,  J/N  and  RF-4B. 

(See  Table  C-6) 

In-House  (North  Island  NARF) 

A- 6  TRAM  DRS 

Detection  and  Ranging  System  of  A-6L  TRAM. 

AN/AAS-33 

Grunman ,  Applied  Science 

A-/  HUD  Test  Bench 

Heads-Up  Display  of  A-71. . 

AN/AVM- 1 1  , 
AN/AVQ-7 

Educational  Computer  Corporation, 
Applimation,  Nonpywell,  Vought, 

AACTS  Engineering 

MA-.l  Test  Bench 

Aircraft  12KVA  generator  test  bench. 

MA-3 

Ai  iil  Imatlon,  Sevi  1 1  e 

I.A-6B  icap-i  tjs 

Tactical  Jaimlng  System. 

AN/AIQ-99 

AN/ALQ-9? 

Grumman 

Navy  Afloat 

1  ft  R 

Integrated  Radio  Room  of  Trident  submarine. 

AN/BSC- 1 

RCA,  Educational  Computer 

Corporation 

CtWb 

Short  range  anti -al rcraf t  qun  system  for 
surface  ships  (Phalanx  close-in  weapon 
system) . 

.. 

Cubic 

Waste  Heat  Boiler 

DO  963  Waste  Heat  Bot ler. 

- 

Applimation 

MK  v  res 

Fire  Control  System  MK  9/,  Mod  (). 

ICS  MK  92  Mod  () 

Sperry 

Air  Torce 

6H83  Test.  Bench 

Test  bench  for  a  portion  of  F-lll  avionics. 

AN/ASM-A27 

Honeywel 1 

AWACS  Navigation 

Navigation  system  of  E-.TA. 

AN/A5N-I18 

Honeywell,  American  Institute 
for  Research 

AWACS  Radar 

Radar  system  of  E-.1A. 

AN/APY- 1 

Honeywell,  American  Institute 
for  Research 

r-16 

Avionics,  electrical,  propulsion,  hydraulic, 
weapon  control  systems  of  F-16. 

-- 

Honeywel 1^ 

rP5-43t 

Ground  radar  system. 

AN/TPS-43E 

In-house  (Keesler  AFB) 

Army 

F i refinder 

Mortar  and  artillery-locating  radar  systems. 

AN/TPQ-36, 

AN/TPQ-37, 

Hughes  Aircraft 

Rrts 

Tactical  communication  system. 

AN/TRC-138. 
AN/TCC- 73  (3), 
AN/TRC-145, 
AN/TRC- 1 51 . 
AN-TSQ-84 , 
AN/TSQ-85 

Gould 

aWhere  more  than  one  contractor  has  been  involved  in  a  program  the  ame  of 

the  principal  contractor  Is  underlined. 

j  Subcontractor  to  the 

weapon  system  contractor. 
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NON-STANDARD  SIMULATOR  PROGRAMS,  CHARACTERISTICS 


TABLE  12.  (Continued) 


1 


These  17  programs  do  not-  seem  to  provide  a  representative 
sample  of  the  potential  range  of  application  for  the  non¬ 
standard  simulators.  All  but  the  three  research  programs 
(A- 7  HUD,  MA-3,  and  6883  Test  Benches)  are  concerned  only 
with  organizational  maintenance,  and  these  three  were  devel¬ 
oped  as  research  vehicles.  Initially,  none  were  envisioned  to 
provide  main-line  training,  although  current  planning  is  for 
the  MA-3  to  provide  training.  Only  four  programs  (MA-3  Test 
Bench,  F-16,  CIWS,  DD  963  boiler)  simulate  other  than  electronic 
equipments.  The  F-16  and  CIWS  are  large  simulation  programs 
to  provide  training  in  several  of  the  skills  (including  elec¬ 
tronics)  required  for  maintenance  of  a  weapon  system.  This 
seems  hardly  representative  of  overall  maintenance  training 
requirements  and  is  quite  different  from  the  pattern  observed 
for  the  standard  systems. 

a.  Concurrent  Development.  The  sample  is  sufficient, 
though,  to  illustrate  some  of  the  characteristics  associated 
with  non-standard  simulators,  three  of  which  are  discussed  here. 
The  timing  of  deliveries  of  training  devices  is  critical  for 
the  introduction  of  new  or  modified  operational  equipments. 
Training  equipment,  of  whatever  type,  must  be  in  place  before 
training  can  commence,  and  personnel  must  be  trained  before 
the  operational  equipment  can  become  an  effective  part  of  the 
force.  Training  simulators  require  their  own  development 
period,  and  this  must  occur  concurrently  with  development  of 
the  operational  equipment.  However,  the  operational  equipment 
is  subject  to  frequent  modification  during  development  and  for 
a  considerable  period  after  its  initial  fielding.  Even  minor 
modifications  can  have  a  large  impact  on  the  costs  of  simulator 
development.  At  least  five  of  the  17  simulator  programs  in¬ 
volved  concurrent  development;  the  A-6  TRAM  DRS ,  the  A- 7  HUD, 
the  Trident  Radio  Room,  the  F-16  trainers,  and  Firefinder.  In 
each  case,  the  simulator  programs  incurred  significant  engineer¬ 
ing  changes  that  increased  their  costs.  The  A-6  program  required 
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extensive  software  changes  that  amounted  to  30  percent  of  the 
final  program  cost.  The  addition  of  FLTR  to  the  A-7  HUD  trainer 
accounted  for  approximately  40  percent  of  the  total  program 
cost.  In  the  F-16  program,  it  seems  ^  .possible  to  attribute  a 
dollar  cost  to  changes,  but  they  are  generally  acknowledged  to 
be  a  major  portion  of  a  cost  overrun  that  amounted  to  three 
times  the  initial  program  estimate.  Not  only  were  changes  to 
the  aircraft  frequent,  but  documentation  of  the  changes  that 
were  necessary  for  simulator  design  ran  as  much  as  12  months 
behind  implementation  of  the  changes  themselves.  Changes  in 
the  Trident  Radio  Room  and  Fire  finder  program  were  not  as  dra¬ 
matic  but  still  had  a  significant  impact  on  development  costs. 

A  related  problem  is  that  modifications  and  configura¬ 
tion  changes  are  common  for  aircraft  that  have  been  fielded  for 
a  considerable  period.  Changes  to  operational  systems  may  re¬ 
sult  in  simulator  modifications  whose  costs  approach  the  cost 
of  development  of  the  original  device.  This  is  close  to  the 
situation  of  the  A-7  HUD  simulator;  a  day  version  of  the  opera¬ 
tional  equipment  had  been  in  use  for  several  years,  and  the  FLIR 
version  entered  development  curing  development  of  the  trainer. 
Contract  cos  attributable  to  modifying  the  trainer  to  simulate 
the  FLIR  amounted  to  85  percent  of  the  original  contracts  for 
the  day  version  trainer.  Modifications  to  operational  equipment 
have  resulted  in  the  obsolescence  (and  discarding  rather  than 
modification'  of  a  number  of  standard  simulators. 

b.  Quantities  Fabricated.  For  nine  programs  in  this 
sample,  for  which  cost  information  was  available,  development 
cost  averaged  over  three  times  the  recurring  cost  of  simulator 
fabrication  and  initial  support.  This  provides  a  large  poten¬ 
tial  for  reducing  average  costs  by  simulation  of  equipment  for 
which  there  is  an  extensive  training  requirement,  such  as  equip¬ 
ment  used  for  general  skill  training  and  equipment  that  is  used 
on  widely  held  weapon  systems.  Only  three  of  the  17  programs  in 
the  sample  simulate  this  type  of  equipment.  The  MA-3  Test  Bench 
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is  a  universal  test  stand  used  throughout  the  Navy  for  onshore 
testing  of  all  models  of  generators  and  constant  speed  drives 
that  supply  aircraft  primary  electric  power;  although  the  MA-3 
Simulator  is  a  research  device,  with  only  one  unit  built,  it 
has  a  potential  for  providing  training  at  all  Naval  and  Marine 
Corps  air  stations.  The  CIWS  and  Mk  92  Fire  Control  System 
are  to  be  installed  on  a  large  number  of  surface  ships,  generat¬ 
ing  an  extensive  training  requirement;  current  planninq  calls 
for  fabrication  of  36  sets  of  eight  simulations  for  CIWS  train¬ 
ing  and  24  sets  of  12  simulations  for  Mk-92  FCS  training.  In 
contrast,  a  few  units  appear  to  satisfy  the  training  requirements 
for  the  bulk  of  the  other  simulators  in  the  sample.  As  examples, 
the  present  F-16  contract  provides  for  delivery  of  equipments 
to  only  three  air  bases;  traininq  for  specific  types  of  Naval/ 
Marine  Corps  aircraft  is  provided  at  only  one  to  three  air  sta¬ 
tions  so  that  buys  of  weapon-specific  simulators  (such  as  the 
A- 6  TRAM  DRS )  will  be  limited  to  a  small  number. 

c.  Substitution  Relations.  Simulators  are  generally 
viewed  as  substitutes  for  actual  equipment  trainers.  Whether 
this  is  a  correct  way  to  view  simulators  should  be  argued 
separately;  simulators  and  actual  equipment  can  each  be  used 
for  traininq  in  ways  that  are  not  possible  by  the  other.  The 
question  of  substitutability  is  not  a  simple  one  and  the  extent 
of  substitution  depends  on  the  nature  of  the  simulation,  the 
equipment  being  Simula _ed ,  and  the  extent  of  training  provided. 
Within  these  14  programs  are  examples  of  four  different  rela¬ 
tions  between  simulators  and  actual  equipment  trainers. 

The  MA-3  Test  Bench  and  6883  Test  Bench  programs  il¬ 
lustrate  cases  approachinq  pure  substitution.  Each  program 
provides  traininq  in  both  the  operation  of  a  test  bench  (i.e., 
maintenance  of  operational  equipment)  and  maintenance  of  the 
test  bench  itself.  Each  simulator  was  designed  to  replace 
some  (but  not  necessarily  all)  operational  equipment  that  had 
been  used  for  traininq. 
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In  the  case  of  the  AWACS  system,  the  navigation  system 
simulator  has  been  used  to  implement  a  substitution  of  school 
training  for  on-the-job  training  (OJT) .  Prior  to  introduction 
of  the  simulator,  formal  (school)  training  was  limited  to  the 
classroom,  and  hands-on  systems  training  was  provided  only  as 
OJT  at  an  operational  base.  Introduction  of  the  simulator 
permitted  the  hands-on  training  to  commence  at  the  training 
school  and  should  result  in  a  shorter  period  of  OJT  before 
personnel  are  qualified  for  independent  work. 

Trident  radio  room  training  employs  both  the  simula¬ 
tors  and  a  modified  complete  operational  radio  room  (actual 
equipment  trainer).  That  is,  they  complement  one  another, 
with  each  contributing  to  different  elements  of  the  curriculum. 

They  are  also  substitutes.  An  early  assessment  of  Trident 
training  requirements  developed  two  alternative  equipment 
configurations  for  radio  room  training.  One  was  the  current 
combination  of  simulation  and  AET.  The  second  was  the  use  of 
three  AETs  only.  The  choice  of  the  combination  of  actual  equip¬ 
ment  and  simulation  was  based,  at  least  in  part,  on  cost  con¬ 
siderations  . 

In  the  case  of  several  other  programs,  both  actual 
equipment  and  simulators  are  used,  but  for  somewhat  different 
reasons.  The  A-6  TRAM  DRS  contains  both  electronic  and  mech¬ 
anical  components.  The  simulator  is  limited  to  training  on 
the  electronic  portion  of  the  system,  and  the  actual  equipment 
is  required  for  the  mechanical  training.  None  of  the  organiza¬ 
tional  echelon  aviation  simulators  can  whol.lv  substitute  for 
operational  equipment.  Typically,  both  organizational  and 
intermediate  maintenance  training  is  provided  at  the  same 
location  and,  frequently,  in  the  same  training  course.  The 
intermediate  level  traininq  will  require  actual  equipment,  but 
normally  in  the  form  of  individual  components  rather  than  an 
integrated  system. 

« 
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The  cost  effectiveness  of  simulation  for  training 
maintenance  skills  depends  upon  its  impact  on  total  training 
costs.  Variations  in  the  substitutability  of  simulators  for 
actual  equipment  trainers  imply  that  simple  comparisons  of  the 
relative  costs  of  the  two  types  of  training  equipment  cannot 
be  taken  as  reliable  guides  to  the  relative  costs  of  training. 
Assessments  of  the  cost  advantage  of  using  simulation  must  be 
based  on  comparisons  of  the  total  costs  of  satisfying  particular 
training  requirements  with  and  without  such  simulators. 

D.  STRUCTURE  FOR  COLLECTING  COST  DATA 

The  set  of  cost  elements  shown  in  Table  13  is  an  initial 
formulation  of  a  functional  cost  structure  for  collecting  data 
to  develop  cost-estimating  relationships  and  other  too] s  for 
assessing  the  costs  of  maintenance  training  simulators.  It  is 
a  mixture  of  elements  that  are  generally  associated  with  other 
types  of  military  equipment  as  well  as  those  that  seem  particu¬ 
larly  relevant  to  processor-driven  simulators;  it  relies  heavily 
on  discussions  with  people  who  have  had  experience  with  simula¬ 
tor  procurements. 

This  cost  element  set  is  incomplete  in  two  ways.  First, 
it  is  at  a  level  of  aggregation  that  may  prove  insufficient  for 
identifying  the  basic  cost  drivers.  Second,  even  at  this  rela¬ 
tively  high  level  of  aggregation,  we  are  uncertain  that  it  is 
fully  specified.  With  our  current  knowledge  regarding  the 
determinants  of  cost,  it  does  not  appear  feasible  to  carry  the 
specification  further. 

This  structure  does,  though,  treat  the  two  important  cost 
characteristics  of  maintenance  training  simulators  evidpr.l  in 
the  data  currently  available  --  the  separation  of  recurring  from 
non-recurring  costs  to  identify  program  development  costs  and 
the  separation  of  software  (and  courseware)  from  other  develop¬ 
ment  costs  to  identify  the  (apparently)  dominant  requirement. 
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Further,  it  is  structured  as  a  matrix.  Several  contractors 
may  be  involved  in  a  single  program;  one  contractor  may  be 
engaged  in  several  (sequential)  contracts  on  a  single  program; 
programs  are  generally  sectioned  into  distinct  and  identifiable 
phases;  and,  changes  in  program  scope  and  statements  of  work 
are  frequent.  In  each  case,  the  types  of  work  and  relations 
among  costs  may  differ  in  systematic  fashion,  and  these  dif¬ 
ferences  should  be  preserved  in  whatever  data  are  collected. 

A  major  problem  with  formulating  a  structure  for  collect¬ 
ing  cost  data  at  this  time  is  the  current  paucity  of  data. 

There  is  no  general  requirement  for  systematic  and  periodic 
reporting  of  all  elements  of  the  costs  incurred  by  contractors 
of  training  equipment.  With  a  single  exception,  standardized 
work  breakdown  structure  •;  (WBS)  for  traininq  equipment  have 
not  been  developed  and  employed;  thus  even  if  contractor  costs 
were  to  be  reported,  there  would  likely  be  incompatabi 1 i tie s 
among  the  data  from  different  programs. 

A  periodic  cost  reporting  system  addressed  to  simulators 
should  be  based  on  a  sinqle  basic  WBS  that  would  be  applicable 
to  a  variety  of  simulator  types  and  other  training  equipments 
and  serve  both  proqram-manaqement  and  cost-assessment  functions. 
The  Army  is  currently  developing  a  general  WBS  for  all  training 
equipment.  It  has  yet  to  be  imposed  on  a  procurement  program, 
and  it  appears  to  be  directed  only  to  cost  assessment.  The  Air 
Force  has  developed  a  WBS  that  has  been  used  for  both  management 
and  cost  assessment,  but  its  application  is  limited  to  flight 
simulators.  There  appear  to  be  significant  differences  between 
these  two  structures,  and  neither  seems  to  satisfy  the  criterion 
of  qeneral  applicability  (e.g.,  neither  appears  to  provide  for 
a  definitive  separation  of  recurring  and  non-recurr inq  costs). 

In  general,  the  program  costs  collected  during  this  project 
(contained  in  Appendix  B)  are  assessed  in  the  format  of  Table  13. 
A  next  step  in  assessing  the  format  (i.e.,  the  adequacy  of  data 
it  displays)  would  be  to  obtain  measures  of  simulator  physical 
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and  performance  characteristics  for  the  current  programs,  to 
test  for  relationships  between  these  characteristics  and  levels 
of  costs  and,  since  this  structure  addresses  only  assessment  of 
costs,  to  reconcile  it  with  the  data  requirements  of  program 
management . 

E.  COSTS  OF  MAINTENANCE  TRAINING  SIMULATORS 

1 .  Standard  Systems 

For  practical  purposes,  the  data  now  available  on  stand¬ 
ardized  systems  are  insufficient  for  analysis  of  cost  and  es¬ 
pecially  for  relating  costs  to  physical  and  performance  charac¬ 
teristics  of  the  trainers.  Almost  all  procurements  are  under 
FFP  contracts  where  formal  documentation  is  typically  limited 
to  performance  specifications  issued  with  the  request  for  pro¬ 
posal  (RFP),  contractors'  technical  proposals,  and  the  contract 
itself.  The  physical  and  performance  characteristics  may 
change,  as  the  result  either  of  contract  negotiations  or  of 
subsequent  contract  modifications,  with  the  corresponding  docu¬ 
mentation  not  being  revised. 

Cost  documentation  is  normally  limited  to  the  line-item 
structure  of  the  contract;  for  the  standard  systems  this  is 
sketchy  at  best  and  can  be  misleading..  A  major  problem  is  that 
contract  line-item  structures  are  in  terms  of  the  products  (or 
deliverables)  that  result  from  the  contract  (e.g.,  trainers, 
data,  contractor  field  services,  conferences).  While  this 
structure  does  provide  useful  information  for  cost  control  and 
management,  it  provides  none  of  the  attributes  of  a  functional 
WBS  necessary  for  evaluation.  The  contract  line  item  "trainers" 
typically  encompasses  over  70  percent  of  a  total  contract  value. 
Within  this  70  percent  are  contained  (or  hidden)  those  cost  dis¬ 
tinctions  that  allow  simulator  and  procurement  program  charac¬ 
teristics  to  be  related  to  program- cost  (e.g.,  between  recurring 
and  non-recurring  costs,  between  development  and  fabrication, 
between  hardware  and  software). 
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Nine  contracts  were  reviewed,  and  the  information  they  con¬ 
tain  is  shown  in  Table  14.  (In  the  discussion  below,  these  con¬ 
tracts  are  referred  to  according  to  the  column  number  of  the 
table.)  This  set  of  contracts  includes  four  models  of  standard¬ 
ized  systems  built  by  three  contractors  and  appears  to  present  a 
representative  sample  of  available  data. 

Individual  contracts  show  a  wide  range  in  the  number  of 
different  types  of  trainers  or  simulation  models  developed  (1 
to  27),  the  number  of  trainers  procured  (2  to  114),  and  their 
averaqe  costs  (from  under  $10,000  to  over  $80,000).  An  impor¬ 
tant  feature  for  assessing  costs  is  that  most  contracts  involve 
development  and  procurement  of  trainers  for  a  number  of  training 
applications  and  several  copies  of  each  type  of  trainer. 

The  line  item  listing  shown  in  Table  14  is  close  to  the  full 
cost  detail  given  in  the  contracts.  The  only  items  contained 
in  the  "other"  cateogry  are  conferences,  training,  and  reliabil¬ 
ity  and  maintainability  programs  and  demonstrations.  A  separate 
line  item  is  normally  shown  for  each  type  of  trainer  delivered 
on  the  contract,  but  that  single  line  will  contain  the  cost  of 
both  the  first  or  prototype  unit  (with  the  development  costs 
it  entails)  and  all  follow-on  units. 

Separation  of  costs  between  the  prototype  and  follow-on 
units  is  contained  only  in  contracts  1,  6,  and  9.  In  contract 
1,  the  same  unit  cost  is  charged  to  all  101  follow-on  units;  in 
contract  9,  follow-on  units  (not  included  in  the  Table  14  values) 
are  specified  as  a  contract  option  at  a  cost  different  from  the 
prototypes,  but  four  of  the  six  trainer  types  are  attributed 
with  the  same  follow-on  unit  cost.  In  three  other  contracts, 
the  same  unit  cost  is  charqed  to  several  different  types  of 
trainers  (both  prototypes  and  follow-ons).  In  contract  1,  the 
average  ratio  of  prototype  to  follow-on  unit  costs  is  approxi¬ 
mately  16  to  1,  while  in  contract  9  the  ratio  averages  3  to  1. 
Since  ECC  is  the  contractor  in  each  case  (although  different 
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TABLE  14.  STANDARD  SIMULATOR  CONTRACT  INFORMATION  AND  COSTS 

(NINE  CONTRACTS) 


Contract3 

1 

2 

3 

4 

5 

6 

7 

8 

9b 

Service 

Navy 

Navy 

Navy 

Navy 

Army 

Army 

Army 

Army 

Army 

r iscal  Year  of  Contract 

1975 

1978 

1979 

1978 

1978 

1977 

1978 

1979 

1979 

Simulator  Model 

ECU 

ECU 

Ridgeway 

Bvrtek 

EC3 

EC3 

EC3 

Ridgeway 

EC3 

Type  of  Contract 

FFP 

FFP 

FFP 

FFP 

FFP 

FFP 

FFP 

FFP 

FPIF 

Contract  Value  (000) 

$1 .13-- 

$1 ,301 

$1,131 

$259 

$552 

$1 ,770 

$1 ,556 

$236 

$2,651 

Number  of  Trainers  Procured 

1 1 4C 

91 

103 

2 

17 

72 

28 

4 

13 

Number  of  Simulation  Models 

Developed 

13 

14 

18 

2 

5 

7 

1 

1 

6 

Average  Number  of  Trainers  of 

Each  Type 

4.2C 

6.5 

5.7 

1 ,0 

3.4 

10.3 

25 

4.0 

2.0 

Averaqe  Contract  Cost  Per  Trainer 
Delivery  (D00) 

$9.9 

$14.3 

$11.0 

$129.5 

$32.5 

$24.6 

$55.6 

$59.0 

$203.9 

Average  Cost  of  Trainers  (000) 

$23.8 

$18.3 

$34.0 

$26.6 

$82. 6d 

Range  of  Unit  Cost  of  Trainers  (000) 

e 

$8.9-16,9 

$6. 5-9.0 

$56  J  61 

Contract  Costs  by  line  Item  (000) 

Trainers  (including  installa¬ 
tion  ) f 

$901 

$1,268 

$805 

$117 

$410 

$1 .363 

$i ,065 

$106 

$1,548 

Technical  Data 

142 

181 

27 

93 

18 

244 

57 

18 

293 

Interim  Support 

19 

15 

75 

17 

60 

70 

304 

4 

73 

factory  Repair  of  Spares 
and  Parts 

5 

45 

20 

10 

75 

Contract  field  Service 

40 

12 

25 

12 

110 

145 

Spares  and  Spare  Parts 

38 

10 

50 

3 

70 

Support  and  Test  Equipment 
and  Tools 

22 

65 

9 

8 

IB 

10 

30 

Logistic  Support  Analysis 

29 

145 

Other 

10 

3 

24 

8 

n 

52 

13 

8 

437'* 

Contract  Total 

1,132 

1  ,482 

1,131 

259 

552 

1,769 

i  ,663 

236 

2,641 

Several  contracts  have  undergone  modifications.  Where  information  was  available, 
j  modifications. 

the  values 

in  this  table  reflect  the 

^Contract  included  one  "hands-on"  trainer  with  a  cost  of  $567 
trainer  is  included  in  the  costs  below,  except  as  noted. 

thousand  and  12  simulation  trainers.  Cost  of  the  hands-on 

Procurement  included  16  ECU  consoles,  13 
simulations  (45  total  devices)  developed 
developed  by  the  contractor  for  the  civil 

different  simulations  (66  total  devices)  developed  on  this  contract,  11  different 
on  an  earlier  contract,  and  three  devices,  addressing  basic  skills,  that  were 
market. 

^Excluding  cost  of  the  hands-on  trainer. 

The  range  of  prototype  (first  unit)  costs  was  $27.9  to  $32.2  thousand.  All  follow-on  units  were  priced  at  $1.8  thousand, 
regardless  of  whether  the  device  was  developed  in  this  contract  or  the  earlier  contract.  Consoles  were  priced  at  $16.5 
thousand  each. 

^Contract  lines  item  listings  normally  show  each  type  of  trainer  and  its  costs  as 
of  all  trainers  of  one  type  will  generally  be  contained  in  that  single  entry. 

a  single  contract  item.  However,  the  costs 

^Includes  $170  thousand  for  claims  resulting  from  contract  modi f ications  and  $241 
system  described  as  "for  test  set(s),  procedure  and  performance  monitoring...". 

thousand  for  extensions  to  the 

software 

standard  systems  are  involved),  we  would  expect  that  the  dis¬ 
tinction  between  recurring  and  non-recurring  functions  would 
be  the  same  for  each  contract.  If  this  is  true,  the  wide  vari¬ 
ations  in  the  recurring/non-recurring  ratios  are  difficult  to 
accept.  In  only  two  contracts  (1  and  6)  are  the  costs  of  main 
frames  and  panels  shown  separately,  and  the  ratio  of  main  frames 
to  panels  varies.  No  contract  provides  for  a  separation  of 
functions  associated  with  development  (e.g.,  hardware,  course¬ 
ware)  except  for  technical  data,  and  in  this  case  a  single  line 
entry  applies  to  all  trainers  included  in  the  contract. 

There  is  a  considerable  difference  in  the  structure  of 
costs  among  the  contracts,  and  it  appears  that  the  meanings  of 
contract  line-item  names  have  not  been  consistently  applied. 

For  example,  contracts  2  and  3  involve  deliveriec  (by  different 
contractors)  of  devices  that  satisfy  the  same  training  applica¬ 
tion  and  have  display  panels  constructed  to  a  single  specifica¬ 
tion.  On  the  basis  of  the  ranges  of  unit  costs  that  are  given, 
it  would  appear  that  ECC  costs  are  higher.  However,  on  the 
basis  of  average  contract  cost  per  delivery  (total  contract 
value  divided  by  the  number  of  trainers  procured)  the  difference 
is  considerably  narrower,  and  it  would  appear  that  a  number  of 
contract  functions  that  are  costed  separately  in  the  Ridgeway 
contract  are  included  under  the  cost  of  the  trainers  in  the 
ECC  contract. 

This  discouraging  assessment  of  available  data  has  been 
reinforced  by  discussions  with  procurement  office  personnel  at 
the  Naval  Training  Equipment  Center  (NTEC) .  Several  have 
expressed  opinions  on  two  points  that  impact  on  the  validity 
of  contract  item  costs.  One  is  that  contract  negotiators  focus 
on  "bottom-1 ine"  (total)  costs  and  that,  within  this  constraint, 
contractor  representatives  will  trade-off  the  amounts  charged 
to  individual  line-items  until  the  relationships  among  them 
look  "reasonable".  The  second  point  is  that  contractors  have 
an  incentive  to  inflate  the  cost  of  simpler  devices  and  to 
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deflate  the  cost  of  the  more  complex  devices.  In  this  way,  as 
contractors  are  successful  in  delivering  the  simpler  devices 
early  in  a  contract,  they  can  speed  up  their  receipt  of  progress 
payments  relative  to  actual  expenses.  One  result  of  these  prac¬ 
tices  is  that  the  relationships  among  different  elements  of 
contract  costs  will  he  distorted,  and  providing  more  detailed 
cost  statements  will  do  little  or  nothing  to  provide  accurate 
relationships  between  physical  and  performance  characteristics 
and  costs. 

2 .  Non-Standard  Systems 

The  program  costs  discussed  in  this  section  are  based  on 
nine  programs  for  which  information  was  either  received  in  or 
translated  to  the  format  shown  in  Table  13.  These  programs 
are  discussed  briefly  and  their  individual  costs  displayed  in 
Appendix  B. 

The  cost  information  comes  from  two  sources.  One  consists 
of  contracts  and  contractors'  proposals;  there  is  no  way  to 
determine  if  the  level  and  structure  of  costs  contained  in 
these  early  estimates  did  occur.  The  second  source  is  program 
office  estimates  of  incurred  costs  based  on  the  records  and  the 
expertise  of  program  office  personnel. 

In  either  case,  there  is  no  way  to  compare  these  estimates 
against  true  costs.  Simulator  programs  fall  below  the  cost 
threshold  of  major  procurements  for  which  contractors  are  re¬ 
quired  to  submit  periodic  reports  in  a  prescribed  WBS .  Contrac¬ 
tors  employ  different  terminologies;  the  structure  of  their 
accounting  systems  differ,  and  there  is  an  ever-present  possi¬ 
bility  of  misinterpretation  in  translating  the  available  infor¬ 
mation  into  the  categories  and  format  shown  in  Table  13.  Con¬ 
sidering  the  wide  range  of  possible  differences  among  simula¬ 
tors  and  simulator  procurement  programs,  we  question  whether  a 
sample  of  nine  programs  is  satisfactory.  However,  it  does  pro¬ 
vide  insights  into  two  important  cost  characteristics  that  are 
discussed  below. 
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Table  15  shows  the  percent  distribution  of  program  total 
cost,  according  to  cost  element,  in  terms  of  the  lowest  and 
highest  observed  percentages  in  this  table  and  the  average  of 
the  percentages.  Note  that  the  percentages  have  been  normal¬ 
ized  in  the  following  two  ways: 

1.  Recurring  production  costs  have  been  adjusted  to  the 
level  of  costs  that  would  have  been  incurred  if  only 
one  unit  had  been  produced  by  dividing  recurring  fab¬ 
rication  costs  by  the  quantity  fabricated.  This  ad¬ 
justment  provides  a  consistent  base  for  the  relation 
between  recurring  and  non-recurring  costs. 

2.  The  MA-3  and  6883  Test  Bench  research  programs  incurred 
significant  costs  for  evaluation  that  were  not  included 
in  calculating  the  test  and  evaluation  percentages  in 
order  that  all  simulator  programs  might  be  treated  as 
though  they  were  intended  for  main-line  training. 

Two  distinct  patterns  emerge  from  this  small  sample.  The 
first  is  the  consistently  high  proportion  of  total  costs  that 
are  devoted  to  the  non-recurring  functions  (primarily  design 
and  development)  when  small  production  quantities  are  involved. 
Further,  the  average  recurring  production  cost  (18  percent)  is 
probably  overstated  as  only  the  AT  Trainer  and  6883  Test  Bench 
programs  identified  the  non-recurring*  portion  of  fabrication 
cost  that,  in  these  cases,  averaged  40  percent  of  the  first 
unit  recurrinq  fabrication  cost. 

Figure  4  is  a  plot  of  the  non-recurring  percentages  when 
program  costs  are  adjusted  only  to  exclude  evaluation  costs 
of  the  MA-3  and  6883.  The  outlying  high  point  is  the  AWACS 
Navigation/Guidance  system  program,  and  there  is  no  explanation 
why  the  percentage  is  this  high.  The  outlying  low  point  is  the 
Visual  Target  Acquisition  System  (VTAS)  program.  A  review  of 


*Tooling,  planning,  and  the  other  requirements  normally  charged 
to  production  accounts  that  do  not  increase  with  quantity. 
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TABLE  15.  PERCENT  DISTRIBUTIONS  OF  COST 
EIGHT  NON-STANDARD  SIMULATOR  PROGRAMS 


BY  CATEGORY  FOR 
(NORMALIZED)3 


Percentage  Distribi 

.it  ions 

Cost  Category 

Lowest 
Observed  in 
Any  Proqram 

Highest 
Observed  in 
Any  Proqram 

Average  of 
Observed 
Percentages 

Non-recurring  Costs 

Front  End  Analysis 

0 

18 

8 

Design  and  Development 

34 

81 

54 

Hardware 

2 

36 

16 

Software/Courseware13 

12 

53° 

31 

Technical  Data 

0 

21 

6 

Hardware  Fabrication  (Non-recurring)^ 

4 

6 

5 

Test  and  Evaluation 

1 

3 

1 

Program  Management 

3 

24 

11 

Total  Non-recurring 

61 

92 

78 

Recurring  Costs 

Production 

5 

38 

18 

Hardware  Fabrication 

4 

36 

15 

Other 

0 

9 

4 

Logistic  Support 

0 

13f 

5 

Initial  Training 

0 

4 

2 

Total  Recurring 

■-  ■  -  —  ,  ■  -  „  . .  -  - - 

8 

39 

25 

aRecurri ng  production  costs  were  adjusted  to  reflect  a  production  quantity  of  one; 
test  and  evaluation  coses  of  the  research  programs  were  not  included. 

bData  on  several  programs  did  not  separate  software  and  courseware  development 
costs.  In  this  table  and  remainder  of  this  section,  these  cost  elements  are 
combined  and  referred  to  as  "software/courseware. " 

cThe  high  percentage  case  is  a  program  that  incurred  software  problems  because  of 
concurrency.  The  next  highest  program  incurred  42  percent  of  total  costs  for 
software/coursewa re, 

^Based  on  two  programs. 

eBased  on  six  programs. 

^The  two  highest  percentages  arose  from  (1)  development  of  a  complete  depot 
maintenance  facility  and  (2)  over  three  years  of  contractor  maintenance 
during  an  extensive  evaluation  program.  The  next  highest  percentage  is  7. 
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U4 

CD 


C9 

ae 

Ui 

a. 


co 

u 

CO 


100 


80 


60 


8-27fi1.11 


QUANTITY  FABRICATED 


KEY:  IRR-1,  TRIDENT  maintenance  trainer 

IRfl-2,  TRIDENT  operatlons/maintenance  trainer 
A  A-6E  DRS,  excluding  engineering  change 
*  Excludes  evaluation  costs 


FIGURE  4.  Non-recurring  cost  as  a  percent  of 
program  total  cost  according  to 
quantity  fabricated 


the  VTAS  simulation  and  features  of  the  procurement  proqram 
suqqests  that  it  is  different  in  some  fashion  from  other  cur¬ 
rent  (and  probably  future)  non-standard  simulator  programs. 

Other  than  the  research  and  in-house  programs,  VTAS  was  initi¬ 
ated  2  years  earlier  than  any  other  non-standard  simulator. 

Also,  the  avionics  system  simulated  was  quite  simple  by  the 
then-current  standards  and  the  training  requirement  was  simi¬ 
larly  simple.  These  considerations  suggest  possible  differences 
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in  the  mix  of  resources  employed  and  the  contractor's  organiza¬ 
tion  and  management  of  the  program.  In  either  case,  the  differ- 
f  tees  would  impact  on  the  percent  distributions. 

Th<  TRAM  DRS  program  incurred  a  large  cost  for  so:v~ 

ware  ehang  s  resulting  from  changes  in  the  operational  hard¬ 
ware.  When  the  chanqe  costs  are  disregarded,  the  percentage 
is  quite  consistent-  v  ‘  :h  the  other  programs.  The  split  be- 
twef  n  recurrin-t  ind  non-recurring  costs  was  available  for  the 
F-16  simulator  program.  The  percentages  associated  with  this 
program  (the  shaded  area)  are  consistent  with  the  pattern  of 
the  otner  six.  it  would  appear  that  production  quantities 
of  five  and  over  are  required  before  recurring  costs  will 
equal  non-recurring  costs. 

The  second  feature  to  emerge  is  the  high  cost  of  develop¬ 
ing  software  and  ourseware.  Within  this  sample,  the  combined 
cos-:  of  software  and  courseware  averages  over  30  percent  of 
total  program  costs  (as  adjusted  to  reflect  production  of  one 
unit)  end  over  40  percent  of  total  design/development  costs 
(with  a  range  from  17  to  72  percent).  When  software/courseware 
costs  are  plotted  against  total  costs  (not  adjusted  for  the 
production  quantity)  no  distinctive  relationships  are  evident 
(Fig.  5). 

Wc  have  no  explanation  for  the  absence  of  an  orderly 
pattern  or  lor  the  wide  range  of  observed  percentages.  This 
small  sample  contains  programs  wi th  diverse  characteristics 
and,  on  a  case-by-case  basis,  a  number  of  reasons  appear 
plausible.  A  likely  reason  for  at  least  part  of  the  range 
of  values  i  differences  in  accounting  practices  among  con¬ 
tractors.  It  is  also  possible  that  our  inability  to  separate 
software  and  courseware  serves  to  obscure  underlying  relation¬ 
ships  that  may  be  present.  Further  data  and  analyses  will  be 
required  to  provide  any  understanding  of  the  determinants  of 
cost.  Considering  the  magnitude  of  the  costs  in  this  sample, 
such  data  and  analyses  are  warranted. 
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KEY:  IRR-1,  TRIDENT  maintenance  trainer 

IRR-2,  TRIDENT  operationx/malntenance  trainer 
A  A-6E  DRS,  excluding  engineering  changes 
*  Excludes  evaluation  costs 


FIGURE  5.  Software/courseware  cost  as  a  percent  of 
program  total  cost,  according  to  quantity 
fabricated 


It  is  unfortunate  that  available  data  do  not  permit  the 
separation  of  software  and  courseware  development  costs  (or, 
for  the  standard  systems,  the  separation  of  software  and  course¬ 
ware  from  hardware  development  and  production  costs).  It  is 
quite  evident,  at  least  for  the  non-standard  systems,  that 
software  and  courseware  are  siqnificant  cost  items.  A  relevant 
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question,  then,  is  how  these  costs  can  be  reduced  for  future 
simulation  systems.  One  promising  avenue  appears  to  be  the 
development  of  a  single  software  system  with  the  following 
attributes:  (1)  it  would  be  non-proprietary  (e.g.,  owned  by 

the  government,  (2)  it  would  permit  courseware  to  be  written 
in  a  high-level  lanquage  that  could  be  composed  and/or  modified 
by  (e.g.)  in-house  subject  matter  experts,  (3)  it  would  be  suf¬ 
ficiently  general  that  its  use  could  be  imposed  as  a  contract 
condition  otr  design  parameter.  The  development  of  the  Ada 
lanquage  may  be  a  large  step  in  this  direction. 

This  concept  is  not  without  problems  of  implementation. 

It  is  only  reasonable  to  expect  contractors  to  resist  use  of 
such  a  non-proprietary  system.  By  definition,  the  standard 
systems  employ  proprietary  standardized  software.  Some  of  the 
contractors  of  non-standard  systems  appear  ti  have  put  efforts 
into  developing  their  own  software  systems.  In  both  cases, 
contractors  have  expended  assets  in  the  development  of  these 
software  paekaqes,  the  values  of  which  would  be  greatly  dimin- 
i shed . 

Figures  4  and  5  focus  on  one  problem  in  identifying  train¬ 
ing  programs  as  candidates  for  simulation.  Judging  by  the  cur¬ 
rent  non-standard  simulator  proqrams,  most  maintenance  simulator 
applications  arise  in  system-specific  training  and  particularly 
in  aviation  training.  Th i  type  of  training  is  generally  pro¬ 
vided  at  a  small  number  of  sites  and  requires  a  limited  number 
of  training  devices,  implying  a  limited  potential  for  quantity 
production  of  a  particular  model  of  training  simulator  and  thus 
a  limited  opportunity  for  reducing  simulator  costs  through  their 
widespread  adoption.  From  a  cost  standpoint,  the  more  promising 
employments  lie  in  the  training  of  qeneral  skills  and  system- 
specific  training  fot.  widely  held  equipments  where  a  relatively 
large  number  of  simulators  can  be  used. 


76 


F.  SUMMARY 


For  assessing  co.  ts  of  maintenance  training  simulators,  it 
is  useful  to  distinguish  between  what  are  defined  as  "standard," 
"non-standard,"  and  "CAI-like"  sytems.  Differences  among  these 
three  classes  lie  in  the  following  areas: 

•  Physical  characteristics, 

•  Complexity  and  cost, 

•  Extent  of  use  within  the  Services,  and 

•  Contracting  practices  employed  for  their  procurement. 

A  standard  system  consists  of  a  standardized  physical  configura¬ 
tion  that  can  he  adapted  to  many  training  applications  through 
courseware  and  pictorial  representations  that  are  tailored  to 
the  particular  eguipment  being  simulated.  Non-standard  systems 
are  typically  unique,  in  total,  for  each  specific  training 
application.  A  CM-like  system  typically  uses  a  2-dimensional 
display  (e.g.,  CRT,  random  access  slide  projector)  to  present 
lesson  materials,  and  can  simulate  different  equipments  through 
courseware  introduced  into  its  computer. 

Available  cost  data  are  not  adequate  for  developing  the 
cost  relationships  necessary  for  comparative  assessments  of 
alternative  ma intenance  training  simulators.  Factors  contribut¬ 
ion  to  this  condition  are  as  follows. 

•  Simulator  programs  fall  below  the  cost  threshold  for 
periodic  reporting  of  incurred  costs  in  a  standard  WBS . 

•  The  data  that  are  available  may  contain  systematic 
biases  so  that  their  reliability  may  be  questioned. 

•  For  the  standard  systems  firm-fixed-price  ( FFP)  contracts 
have  been  prevalent;  the  only  generally  available  cost 
information  is  limited  to  precontract  documentation  and 
the  contracts  themselves. 

•  Within  the  small  number  of  non-standard  systems  that 
have  been  built,  there  has  been  a  wide  range  of  program 
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arrangements  and  purposes,  device  complexity  and  char¬ 
acteristics,  and  training  capabilities.  It  is  doubtful 
whether  this  small  number  would  provide  a  satisfactory 
base  for  developing  a  cost-analysis  capability. 

•  For  practical  purposes,  cost  data  on  CAl-like  maintenance 
simulator  system  costs  are  not  available.  Only  one 
experimental  system  has  been  built,  and  contracts  for 
prototype  development  of  two  other  systems  have  only 
recently  been  let. 

There  is  an  obvious  advantage  in  the  concurrent  develop¬ 
ment  of  operational  and  training  equipments  so  that  trainers 
are  available  at  the  time  operational  equipment  is  first 
fielded.  However,  this  practice  entails  risk  since  the  opera¬ 
tional  equipment  is  subject  to  continual  change  and  even  minor 
changes  may  result  in  high  cost  modifications  to  training 
simulators  (especially  to  the  simulation  software  and  course¬ 
ware  ) . 

For  the  non-standard  simulators,  non-recurring  costs 
account  for  the  major  portion  of  total  costs  when  production 
quantities  are  small  (e.g.,  five  or  less).  However,  most 
potential  applications  appear  to  be  in  weapon-system-specific 
training  (especially  in  aviation)  where  a  limited  number  of 
devices  would  be  required. 

For  the  non-standard  simulators,  software/courseware 
(i.e.,  program  design  and  programming)  appears  to  be  the 
single  largest  element  of  cost.  Where  cost  overruns  have 
occurred,  they  appear  to  have  been  primarily  due  to  software 
development  problems.  There  should  be  a  significant  cost 
advantage  gained  by  development  of  a  widely  applicable  and 
non-proprietary  software  system. 
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IV.  COST-EFFECTIVENESS  OF  MAINTENANCE  SIMULATORS 

The  crucial  question  is  whether  maintenance  simulators  are 
cost-effective  for  training  military  technicians.  Since  cost, 
effectiveness,  and  cost-effectiveness  are  not,  in  themselves, 
absolute  quantities,  this  question  must  be  answered  in  relative 
terms,  i.e.,  compared  to  what  el se  is  a  particular  maintenance 
simulator  cost-effective?  All  of  the  studies  with  relevant  data 
compared  the  cost  and  effectiveness  of  maintenance  simulators  to 
that  of  actual  equipment  trainers. 

With  respect  to  effectiveness,  the  data  from  12  studies 
show  that  student  achievement  at  school  is  about  the  same  for 
those  trained  with  simulators  as  for  those  trained  with  actual 
equipment  trainers;  there  was  one  case  where  students  trained 
with  simulators  had  poorer  achievement  scores.  We  would  prefer 
to  estimate  the  effectiveness  of  maintenance  simulators  and  of 
actual  equipment  trainers  by  comparing  the  performance  of  tech¬ 
nicians  (trained  with  one  or  the  other)  on  the  job  rather  than 
just  at  school.  Job  performance  could  be  measured  by  data  such 
as  the  time  needed  to  identify  malfunctions  and  to  repair  or 
replace  faulty  components,  the  number  (or  percent)  of  repairs 
where  good  parts  were  removed  unnecessarily  or  bad  parts  not. 
identified  and  so  on. 

No  evaluation  of  a  maintenance  simulator  reported  objective 
job  performance  data.  In  one  study,  supervisors'  ratings  (i.e., 
subjective  data)  showed  about  the  same  level  of  job  performance 
for  technicians  trained  with  the  6883  Test  Station  3 -di mens iona 1 
maintenance  simulator  or  the  actual  equipment  (Cicchinelli  et  al . 
19U 0 )  .  Based  on  the  data  on  student  achievement  at  school  and 
the  one  case  of  supervisors'  ratings  of  on  the  job  performance, 
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we  conclude  that  maintenance  simulators  and  actual  equipment 
trainers  are  equally  effective  for  training  maintenance  tech¬ 
nicians  . 

This  finding  is  based  on  a  wide  spectrum  of  simulators, 
i.e.,  maintenance  simulators  of  radars,  vehicles,  electromechan¬ 
ical  equipment,  2-D  and  3-D  designs,  and  simulators  that  are 
used  for  training  organizational  and  intermediate  maintenance. 

It  would  be  tempting  to  infer  that  one  type  of  simulator  or  a 
particular  way  of  using  them,  among  these  classes,  is  more  ef¬ 
fective  than  another.  No  such  breakdown  appears  possible  with 
the  limited  data  available.  We  cannot  answer  such  interesting 
questions  as  how  effectiveness  might  vary  with  cost  or  how 
cost  might  vary  with  effectiveness,  because  no  such  trade-offs 
have  been  undertaken.  We  have  only  one-point  comparisons  of  the 
costs  of  maintenance  simulators  and  of  actual  equipment  trainers 
that  have  been  shown  to  have  equal  effectiveness  for  training 
at  school.  So,  we  are  left  with  the  gene. al  conclusion,  as 
stated  above,  that  maintenance  simulators  and  actual  equipment 
trainers  are  equally  effective  for  training  technicians. 

Our  evaluation  of  costs  uses  the  cost  data  presented  in 
Chapter  III?  these  describe  acquisition  but  not  life-cycle 
costs.  The  costs  of  acquiring  actual  equipment  or  simulators 
do  not  include  the  costs  of  their  use  for  training  purposes, 
e.g.,  the  operating  costs  of  training  such  as  instructors, 
student  pay  and  support,  maintenance  of  training  equipment, 
and  management  of  the  school.  A  cost-effectiveness  evaluation 
based  on  acquisition  costs  alone  must  be  regarded  as  incomplete 
compared  co  one  that  includes  all  life-cycle  costs.  A  single 
exception,  in  the  case  of  the  life-cycle  cost  comparison  of 
the-  6883  Test  Stand  3-D  simulator  and  actual  equipment  trainer, 
reported  by  Cicchinelli  et  al.  (1980),  will  be  considered 
separate  1 y . 

Table  16  shows  the  acquisition  co.ets  of  comparable  simu¬ 
lators,  actual  equipment  trainers,  and  operational  equipment 
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ACQUISITION  COSTS  OF  COMPARABLE  MAINTENANCE  SIMULATORS,  ACTUAL  EQUIPMENT 
TRAINERS  AND  OPERATIONAL  EQUIPMENT  NOT  CONFIGURED  FOR  TRAINING 


(see  next  page  For  footnotes) 


TABLE  16.  (Continued) 


(before  modification  for  use  in  training)  in  20  maintenance 
simulate,  programs.  It  is  important  to  understand  the  different 
types  of  cost  data  shown  in  this  Table. 

Operational  Equipment  Unit  Cost  -  Production  cost  of  an 
additional  unit  of  equipment  designed  to  meet  some 
military  purpose;  these  values  do  not  include  RDT&E 
costs.  Where  the  costs  of  an  actual  equipment  trainer 
are  not  available,  these  costs  are  used  as  a  proxy 
for  actual  equipment  cost. 

Actual  Equipment  Trainer  Unit  Cost  -  Cost  of  operational 
equipment,  immediately  above,  that  has  been  adapted 
for  use  in  training,  e.g.,  power,  special  inputs  and 
controls,  etc.  Such  modifications  require  additional 
costs . 

Simulator  Total  Program  Cost  -  Cost  of  RDT&E,  prototypes, 
and  manufacturing  facilities  needed  to  produce  one 
or  more  maintenance  simulators.  In  our  sample  of  20 
simulator  programs  for  which  total  program  cost  data 
could  be  compiled,  there  were  12  instances  in  which 
only  one  simulator  was  built;  in  the  eignt  other 
programs,  from  2  to  36  units  were  built. 

Simulator  Normalized  Total  Program  Cost  -  Total  production 

costs  adjusted  to  reflect  a  production  quantity  of  one; 
includes  the  costs  of  research  and  development  but  not 
test  and  evaluation  of  simulators  developed  in  research 
programs . 

Simulator  Unit  Recurring  Fabrication  Cost  -  The  cost  of 
producing  a  follow-on  unit  of  equipment  after  the 
costs  of  RDT&E,  prototypes,  and  manufacturing  facil¬ 
ities  have  been  accounted  for.  This  is  a  maximum 
value;  where  data  did  not  provide  a  separation  between 
recurring  and  non-recurring  production  costs,  the 
value  shown  is  total  production  cost.  No  value  is 
shown  where  cost  data  did  not  allow  an  estimate  of 
production  cost. 

8  3 


There  are,  thus,  several  ways  to  compare  the  costs  of  ac¬ 
quiring  maintenance  simulators  and  actual  equipment  trainers. 

The  cost  of  an  actual  equipment  trainer  is  set,  approximately, 
by  the  incremental  cost  of  procuring  one  additional  unit  of 
operational  equipment  plus  the  cost  of  any  modification  neces¬ 
sary  for  its  use  in  a  classroom.  This  is  a  unit  recurring 
fabrication  cost,  devoid  of  RDT&E  and  non-recurring  production 
costs.  We  have  these  costs  for  six  AETs  Where  AETs  have  not 
been  built,  wo  can  use  the  operational  equipment  unit  cost  a 
a  surrogate  AET  cost  for  comparing  the  costs  of  simulators  and 
AETs . 

The  average  ratio  of  AET  unit  cost:  operational  equipment 
unit  cost  is  1.27;  that  is,  AETs  cost,  on  the  average,  about 
25  percent  more  than  operational  equipment  before  the  latter 
is  modified  for  training  (Table  17);  the  data  are  based  on  only 
five  cases.  These  ratios,  which  vary  from  1.00  to  1.59,  presum¬ 
ably  relate  to  the  degree  of  modification  involved  in  the  vari¬ 
ous  cases;  whether  further  modification  at  even  greater  cost 
would  improve  the  effectiveness  of  instruction  has  not  been 
examined . 

We  will  estimate  the  cost  of  acquiring  a  maintenance  simu¬ 
lator  in  two  ways.  The  first  estimate  includes  non-recurring 
costs  (e.g.,  research,  development,  and  manufacturing  facilities) 
and  the  costs  of  manufacturing  one  unit.  This  value  is  the  nor¬ 
malized  total  program  cost,  as  defined  above.  The  second  esti¬ 
mate  includes  only  the  unit  recurring  fabrication  cost  (as  de¬ 
fined  above;,  i.e,  the  cost  to  produce  an  additional  unit 
(after  research,  development,  and  other  non-recurring  functions 
have  been  accomplished).  Each  of  these  estimates  is  relevant  be¬ 
cause  of  the  large  disparity  between  recurring  and  non-recurring 
costs.  Maintenance  simulator  programs  have  typically  involved 
small  quantities  so  that  relative  cost-effectiveness  of  their 
use  will  vary  areatly  with  quantity  procured.  To  the  extent 
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TABLE  17.  COMPARISON  OF  THE  ACQUISITION  COSTS  OF 
ACTUAL  EQUIPMENT  TRAINERS  AND  COMPARABLE  OPERATIONAL  EQUIPMENT 
BEFORE  IT  WAS  CONFIGURED  FOR  TRAINING 


Unit  Cost 

(thousands  of  dollars) 

Operational 

Equipment 

Actual 

Equipment 

Trainer 

Cost  Ratio: 
AET/Operational 
Equipment 

MA-3 

110 

175 

1.59 

Trident  Integrated 

Radio  Room 

12,100 

17,500 

1.45 

Trident  High  Pressure 

Air  Compressor 

315 

400 

1.27 

Trident  Air  Conditioner 

530 

550 

1.04 

F-lll  Avionics  Test 

Bench  (6883  Test  Stand) 

1,955 

1 ,955 

1.00 

Mean  1.27 

permitted  by  the  data,  we  have  estimated  the  recurrinq  costs 
even  if  only  one  unit  was  actually  fabricated. 

The  actual  equipment  trainers  and  operational  equipment 
shown  in  Table  16  vary  widely  in  cost  (from  $45,000  to 
$17,500,000).  Therefore,  we  have  used  ratios  to  compare  the 
costs  of  simulators  and  actual  equipment  trainers.  The  central 
tendencies  of  the  cost  ratios,  for  both  the  normalized  total 
proqram  and  unit  recurrinq  estimates,  are  shown  at  the  bottom 
of  Table  16. 

Plots  of  the  individual  ratios  of  estimated  simulator 
costs  (both  recurrinq  fabrication  and  normalized  proqram)  to 
actual  equipment,  trainer  and  operational  equipment  costs  are 
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shown  in  Figs.  6  and  7.*  In  both  figures  the  operational  equip¬ 
ment  costs  have  been  adjusted  by  the  average  ratio  of  costs  de¬ 
veloped  in  Table  17.  With  two  exceptions,  the  recurring  fabri¬ 
cation  costs  of  simulators  (Fig.  6)  are  20  percent  or  less  of 
the  costs  of  either  operational  equipment  (as  adjusted)  or 
actual  equipment  trainers,  and  this  conclusion  does  not  depend 
upon  including  operational  equipment  in  the  sample.  Nine  of 
the  11  cases  (80  percent)  fall  below  this  arbitrary  threshold, 
but  there  is  a  large  dispersion  among  them  ranging  from  3  to 
19  percent.  Available  data  provide  no  explanation  for  this 
range.  The  available  data  provide  some  insight  into  the  two 
cases  that  fall  abo  -t  20  percent.  The  VTAS  simulates  avionics 
equipment  that  has  been  out  of  procurement  for  many  years,  and 
wo  suspect  that  the  cost  of  the  operational  equipment  is  seri¬ 
ously  underestimated.  The  MA- 3  is  a  research  device  and  may 
contain  features  that  serve  only  the  research  function.  How¬ 
ever,  it  does  not  appear  reasonable  that  these  special  features 
alone  would  account  for  its  relatively  high  cost. 

The  relationship  between  simulator  normalized  program  costs 
and  the  costs  of  actual  equipment  trainers  or  operational  equip¬ 
ment  are  not  as  clear-cut  (Fig.  7).  In  seven  of  the  11  cases 
the  simulator  cost  is  less  than  60  percent  of  the  cost  of  the 
actual  equipment  trainer  or  operational  equipment  (with  a  range 
of  2  8  to  53  percent) .  However,  in  the  other  four  cases,  the 
percentages  range  from  160  to  400.  At  first  appearance,  this 
sample  seems  to  come  from  two  populations,  but  we  can  find  no 
support  for  this  argument  in  th°  characteristics  of  either  the 
simulators  or  the  procurement  programs.  Similar  to  the  previous 
ratios  of  recurring  (fabrication)  to  actual  equipment  costs, 


*The  Trident  IRR  maintenance  trainer  has  been  excluded  from 
this  analysis  as  it  appears  to  be  as  much  a  complement  to  as 
a  substitute  for  either  the  actual  equipment  trainer  or  the 
operations/maintenance  trainer. 
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FIGURE  6.  Relation  between  actual  equipment  trainer  and 
simulator  recurring  fabrication  costs 
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the  VTAS  and  MA-3  simulators  are  "outliers"  here  too,  and  we 
suspect  for  the  same  reasons.  The  AT  Trainer  also  simulates 
avionics  equipment  that  has  been  out  of  production  for  a  number 
of  years,  and  we  suspect  the  cost  of  the  operational  equipment 
is  considerably  underestimated.  We  have  no  explanation  for  the 
relatively  high  cost  of  the  AWACS  Navigation/Guidance  Simulator. 
The  contractor  of  this  program  incurred  a  significant  (and  non¬ 
reimbursed)  overrun  that  has  been  attributed  to  his  independent 
development  of  a  courseware  translation  system  (discussed  in 
Appendix  B) .  Even  when  the  overrun  is  subtracted,  the  normal¬ 
ized  program  cost  still  exceeds  the  full  adjusted  cost  of  the 
operational  equipment  by  close  to  25  percent. 

Note  that  three  of  the  four  cases  with  relatively  high 
cost  ratios  involve  the  comparisons  with  adjusted  operational 
equipment  costs,  and  again,  it  appears  that  the  sample  comes 
from  two  populations.  For  example,  in  80  percent  of  the  cases 
where  ratios  are  based  on  actual  equipment  trainers,  the  simu¬ 
lator  normalized  program  cost  is  less  than  50  percent  of  the 
unit  cost  of  the  actual  equipment  trainer;  for  those  ratios 
based  o:i  adjusted  operational  equipment  costs,  the  normalized 
program  cost  is  greater  than  50  percent  of  the  unit  operational 
equipment  cost  in  80  percent  of  the  cases.  We  can  find  no 
rationale  for  this  observation.  No  such  distinction  can  be 
made  with  respect  to  simulator  recurring  fabrication  cost,  and 
we  feel  it  is  spurious. 

The  cost-effectiveness  of  a  maintenance  simulator  on  a 
life-cycle  basis  has  been  evaluated  only  in  one  case,  that  of 
the  Air  Force  (1883  Test  Stand  3-dimensional  simulator  and  actual 
equipment  trainer  ( Cicch inel 1 i ,  Harmon,  Keller,  et  al.,  1980). 

In  a  later  study,  these  authors  will  also  evaluate  a  2-dimen¬ 
sional  version  of  this  simulator.  The  3-dimensional  simulator 
and  actual  equipment  trainer  were  equally  effective  when  meas- 
ured  by  student  achievement  at  school;  supervisors1  ratings 


showed  no  difference  between  the  job  performance  of  students 
trained  either  way  for  periods  up  to  32  weeks  after  leaving 
school . 

The  life-cycle  cost  comparison  cf  .»'•  and  actual 

equipment  trainer  is  shown  in  Table  l  .  -■  v, re 

in  constant  1978  dollars  over  a  15-year  P'  ■  •  discounted 

at  10  percent.  The  results  show  that  t:  -  *  cc  ..  student  hour 

was  $23  for  the  simulator  and  $60  for  t  v  aci.uai  equipment 
trainer,  i.e.,  38  percent  as  much  for  the  simulator,  compared 
to  the  actual  equipment  trainer,  for  all  costs  over  a  15-year 
period.  The  simulator  cost  less  to  procure  ($594,000  vs 
$2,104,000,  or  28  percent  as  much)  and  less  to  operate 
($1,588,000  vs  $3,367,000  or  47  percent  as  much)  over  a  15-year 
period . 

Using  net  present  value  ^  1978  constant  dollars),  the  recur 
ring  costs  were  $1,791,000  or  85  percent  of  the  non-recurring 
costs  of  the  actual  equipment  trainer.  The  recurring  costs  of 
the  simulator  were  $906,000  or  152  percent  of  its  non-recurring 
costs . 

We  draw  the  following  conclusions: 

Cost:  Maintenance  simulators  cost  less  than  actual  equip¬ 

ment  trainers.  On  the  average,  to  develop  and  fabricate 
one  simulator  costs  less  than  60  percent  of  the  cost  of 
an  actual  equipment  trainer;  to  fabricate  one  unit  of  a 
simulator  (once  .  has  been  developed)  costs  less  than  20 
percent  of  the  cost  of  an  actual  equipment  trainer.  How¬ 
ever,  there  is  a  large  dispersion  about  these  averages. 
Effectiveness :  Achievement  at  school  is  the  same  whether 

students  are  trained  with  maintenance  simulators  or  with 
actual  equipment  trainers.  This  finding  applies  to  12  out 
of  13  cases  in  which  such  comparisons  were  made.  There¬ 
fore,  maintenance  simulators  are  cost-effective  compared 
to  actual  equipment  trainers. 


TABLE  18.  LIFE-CYCLE  COST  COMPARISON  OF  6883  TEST  STAND 
ACTUAL  EQUIPMENT  TRAINER  AND  3-DIMENSIONAL  SIMULATOR3 


Costs 

(thousands  of  dollars) 

Cost  Category 

Actual  Equipment  Trainer 

3-01mens tonal 

Non¬ 

recurring 

Costs 

r— 

Recurring 

Costs 

Non¬ 

recurring 

Costs 

Recurring 

Costs 

.. 

Facilities 

1 

110 

1 

no 

Equipment 

2104 

2798 

594 

1000 

Instructional 

materials 

28 

26 

Instructors  and 
overhead 

73 

94 

Students  and 
support 

358 

358 

Total 

2105 

3367 

595 

1588 

Grand  Total 

5472 

2183 

(Net  present 
value,  1978) 

(3896) 

(1501) 

Cost  per  student 
hourb 

$60 

$23 

Estimated  based  on  15-year  life  cycle  discounted  at  10  percent,  in  1978 
constant  dollars,  Modified  from  data  presented  in  Cicchinelli,  Harmon, 
Keller,  et  al . ,  1980,  p.  67-69.  Table  corrected  to  snow  cost  of  instruc¬ 
tors  for  simulator  and  cost  per  student  hour  over  a  15-year  period  for 
AET  and  simulator,  based  on  discussion  with  senior  author.  Analysis 
assumes  720  instructor  hours  per  year  and  operation  of  equipment  for 
2.1  shifts  per  day  to  handle  student  load. 

b180  students  per  yr  x  3  days  per  student  x  8  student  hrs  per  day  x  15 
yrs  =  64,800  total  student  hours. 


This  finding  is  necessarily  qualified  by  the  limited  nature 
of  the  data  from  which  it  is  derived.  Effectiveness,  as  used 
here,  is  based  on  performance  demonstrated  at  school  rather  than 
on  the  job.  Cost,  as  used  here,  refers  to  the  initial  costs  of 
acquiring  training  equipment  and  does  not  include  the  costs 
associated  with  the  operation  of  simulators  or  of  actual  equip¬ 
ment  for  training,  e.q.,  maintenance  and  upkeep,  instructors  and 
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support  personnel,  student  time,  and  the  like.  In  the  one  case 
where  a  life-cycle  cost  comparison  was  made,  total  costs  over  a 
15-year  period  for  the  6883  Test  Stand  3-dimensional  simulator 
was  38  percent  as  much  as  for  the  actual  equipment  trainer. 

Both  were  equally  effective,  as  measured  by  tests  at  school  and 
by  supervisors'  ratings  on  the  job  after  school. 

Insufficient  information  is  available  with  which  to  draw 
conclusions  as  to  whether  2-D  simulators  are  cost-effective  com¬ 
pared  with  3-D  simulators,  the  aspects  of  maintenance  training 
for  which  simulators  are  most  effective,  and  how  to  allocate 
the  amount  of  time,  for  greatest  cost-effectiveness,  between 
maintenance  simulators,  actual  equipment  trainers,  and  on-the- 
job  training.  All  of  these  topics  are  matters  for  further 
research,  development,  test,  and  evaluation  that  are  discussed 
next  in  this  paper. 
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V.  DISCUSSION 


On  the  basis  of  data  presented  in  previous  chapters,  we 
find  that  maintenance  simulators  are  cost-effective  compared 
with  actual  equipment  trainers.  Soth  are  about  equally  effec¬ 
tive  for  training  maintenance  technicians  at  schools;  in  general, 
maintenance  simulators  cost  less  to  acquire  than  do  actual  equip¬ 
ment  trainers.  In  this  chapter,  we  wish  to  discuss  the  signi¬ 
ficance  of  these  findings,  the  limitations  of  the  data  upon 
which  these  findings  are  based,  and  the  steps  that  should  be 
taken  both  the  improve  our  knowledge  and  to  increase  the  cost- 
effectiveness  of  maintenance  simulators  used  to  support  train¬ 
ing  for  future  systems. 

A.  EFFECTIVENESS 

Students  trained  on  maintenance  simulators  perform  as  well 
on  tests  at  school  as  do  students  trained  on  actual  equipment. 
This  finding  is  consistent  with  results  of  studies  with  use  of 
computer-based  instruction  for  technical  courses  on  electricity, 
electronics,  vehicle  repair,  precision  measuring  equipment,  and 
weapons  mechanic  (i.e.,  not  maintenance  training  per  se). 

We  would  expect  that  individualized,  self-paced  instruc¬ 
tion,  an  inherent  characteristic  of  maintenance  simulation, 
would  save  some  of  the  time  students  need  to  complete  instruc¬ 
tion  given  with  equipment,  particularly  where  the  actual  equip¬ 
ment  trainers  are  used  more  for  classroom  demonstration  than  for 
individua’  practice  by  students.  Only  three  studies  of  mainte¬ 
nance  simulators  report  data  on  the  time  needed  by  students  to 
complete  their  courses  (Parker  and  De  Pauli  1967;  Rigney,  Towne , 
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King,  et  al.,  1978;  and  Swezey  1978).  Here,  students  saved  22, 
50,  and  50  percent,  respectively,  of  the  time  needed  previously 
in  courses  given  with  actual  equipment  trainers.  Time  savings, 
if  any,  when  maintenance  simulators  are  evaluated,  should  be 
recorded  in  future  studies;  it  is  anticipated  that  the  few  re¬ 
sults  reported  so  far  will  be  confirmed.  It  is  important  to 
understand  that  all  finuings,  although  positive  with  respect 
both  to  simulators  and  computer-based  instruction,  apply  only 
to  training  at  school;  there  are  no  data  about  the  effect  that 
such  training,  including  that  using  actual  equipment,  may  have 
on  job  performance  in  the  field.  The  importance  of  collecting 
information  about  maintenance  performance  on  the  job,  and  relat¬ 
ing  it  to  method  of  training  at  school,  can  hardly  be  overempha¬ 
sized. 

B.  COST 

Maintenance  simulators  appear  to  cost  less  to  procure  than 
do  actual  equipment  trainers.  With  some  exceptions,  the  cost  to 
develop  and  fabricate  one  simulator  is  less  than  60  percent  of 
the  cost  cf  an  actual  equipment  trainer,  and  to  fabricate  one 
unit  of  a  developed  simulator  is  less  than  20  percent  of  the 
cost  of  an  actual  equipment  trainer.  This  finding  is  based  on 
11  cases  where  meaningful  cost  comparisons  could  be  devised. 
Because  of  the  limited  number  of  cases,  no  attempt  was  made  to 
investigate  the  determinants  of  cost. 

It  is  important  to  emphasize  that  these  comparisons  are 
based  only  on  procurement  costs;  they  are  not  life-cycle  costs. 
Simulators  and  actual  equipment  trainers  are  used  for  training 
over  relatively  long  periods  of  tine  such  as,  for  example,  10 
years.  In  addition  to  the  costs  of  acquisition,  they  incur  costs 
for  operators,  maintenance,  instructors,  and  students.  There¬ 
fore,  life-cycle  costs  are  more  significant  than  acquisition 
costs  alone  as  a  basis  for  evaluating  the  costs  of  alternative 
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training  devices.  There  is  one  recent  estimate  that  the  life- 
cycle  costs  of  a  training  program  using  a  maintenance  simulator 
(the  6883  Test  Station)  would  be  about  40  percent  that  of  one 
using  the  actual  eguipment  (Cicchinelli ,  Harmon,  Keller,  et  al., 
1980) . 

However,  the  cost  data  that  are  now  available  are  not  ade¬ 
quate  for  definitive  conclusions  regarding  the  cost-effective¬ 
ness  of  simulators  vis-a-vis  actual  equipment  trainers.  Some 
of  the  data  that  are  available  appear  to  contain  such  systematic 
biases  that  their  reliability  may  be  questioned.  The  way  in 
which  maintenance  simulators  are  procured  appears  to  contribute 
to  the  inadequacy  of  currently  available  cost  data: 

•  The  cost  of  simulator  programs  fall  below  the  cost 
threshold  of  major  procurements,  with  their  associate 
requirements  for  use  of  a  standard  work  breakdown 
structure  (WBS)  and  for  contractor  cost  reporting 
within  the  WBS  structure.  Contract  line  item  listings, 
that  might  serve  as  a  functional  cost  structure,  vary 
considerably,  both  among  the  Services  and  among  separate 
contracts  within  a  sinqle  Service,  with  a  result  that 
cost  documentation  may  not  be  comparable  among  contracts 

•  Most  maintenance  simulators  with  standardized  software 
systems  have  been  procured  by  means  of  firm-fixed-price 

( FFP)  contracts.  Here,  the  only  costs  that  are  general) 
available  are  limited  to  those  spelled  out  in  the  con¬ 
tract  itself.  For  the  systems  with  non-standard  soft¬ 
ware,  fixed-price-incentive-fee  (FPIF),  and  cost-plus 
fixed  fee  (CPFF)  contracts,  and  CPIF  have  also  been 
employed.  However,  FPIF  contracts  provide  the  Services 
with  little  leverage  in  requiring  contractors  to  pro¬ 
vide  cost  information;  none  of  the  program  offices  that 
have  employed  cost-plus  contractors  have  required  con¬ 
tractors  to  provide  this  type  of  data. 
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Some  of.  the  maintenance  simulators  for  which  cost  data 
are  available  were  procured  primarily  for  purposes  of 
research  and  development.  In  these  cases,  both  the 
programs  and  the  resulting  devices  contain  features  that 
would  not  be  present  had  the  devices  been  intended  only 
for  routine  training.  Costs  for  features  peculiar  to 
research  may  be  considerable  but,  in  general,  they  can¬ 
not  be  identified  and  separated.  Thus,  such  cost  data 
probably  contain  an  upward  bias. 

We  believe,  but  cannot  document,  that  currently 
available  cost  data  on  maintenance  simulators  must  be 
qualified  even  further  for  the  following  reasons: 

1.  Contractors  of  some  systems  with  non-standard 
software  appear  to  have  incurred  losses  that 
(in  whole  or  in  part)  they  have  not  divulged. 
This  would  introduce  a  downward  bias  in  the 
available  data.  This  judgement  has  been  offered 
by  personnel  in  the  program  offices  involved. 

2.  Contracts  for  standardized  systems  typically 
encompass  procurement  of  several  different 
simulators;  negotiations  appear  to  focus  on 
total  contract  cost.  This  has  two  impacts: 

(1)  it  allows  trade-offs  among  individual  con¬ 
tract  line-item  costs  in  order  that  the  rela¬ 
tionships  amonq  them  appear  "reasonable"  to  the 
government;  (2)  contractors  have  an  incentive 
to  inflate  the  costs  of  simpler  devices  and  de¬ 
flate  the  costs  of  more  complex  devices.  The 
result  is  to  distort  the  cost  relationships 
among  contract  elements. 

3.  The  market  appears  highly  competitive  for  sim¬ 
ulators  with  both  standard  and  non-standard 
software,  and  it  ir  difficult  to  get  contrac¬ 
tors  to  provide  detailed  cost  data. 
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4.  Within  the  small  sample  of  non-standard  systems, 
there  exists  a  wide  range  of  program  arrange¬ 
ments,  device  complexity,  physical  characteris¬ 
tics,  and  training  capabilities.  Considering 
this  wide  spectrum,  it  is  questionable  whether 
the  sample  provides  a  sufficient  base  for  devel¬ 
oping  a  cost  analysis  capability,  even  in  the 
absence  of  the  cost  data  problems  discussed 
above . 

C.  COST-EFFECTIVENESS 

Maintenance  simulators  appear  to  be  cost-effective  compared 
to  actual  equipment  trainers  for  training  technicians.  Since 
the  qualifications  that  apply  to  this  finding  have  been  explained 
above,  they  will  only  be  cited  here:  the  finding  is  based  on 
acquisition  rather  than  life-cycle  costs,  on  effectiveness  as 
measured  by  the  performance  of  students  at  school  rather  than 
on  the  job  and  only  on  a  limited  number  of  cases  (N=ll).  Here, 
we  will  try  to  explain  what  this  finding  does  and  does  not  tell 
us . 

We  can  realize  the  cost  advantages  of  maintenance  simula¬ 
tors  only  by  using  them  instead  of  actual  equipment  trainers 
(among  other  training  resources)  in  our  maintenance  training 
programs.  This  is  likely  to  cause  some  problems  for  those  who 
believe  that,  even  if  maintenance  simulators  are  used,  it  is 
still  necessary  to  use  actu-1  equipment  trainers  at  school  to 
train  technicians  how  to  work  later  on  actual  equipment  on  the 
job.  This  dilemma  can  be  resolved  by  comparing  on-the-job  per¬ 
formance  of  those  trained  at  school  only  with  different  mixes 
of  both  actual  equipment  trainers  and  simulators.  An  evaluation 
of  on-the-job  performance  has  been  reported  in  only  one  instance 
(Cicchinelli  et  al.  1980).  Here,  supervisors'  ratings  showed 
no  difference  in  the  job  performance  of  technicians  trained 
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only  with  the  6883  Converter/Fl ight  Control  System  Test  Station 
3-aimensional  Simulator  or  with  the  actual  equipment  trainer. 
Additional  studies  of  this  type  would  be  most  welcome. 

Suppose  that  on-the-job  performance  turns  out  to  be  the 
same  for  students  trained  only  with  simulators  or  with  a  com¬ 
bination  of  simulators  and  actual  equipment  trainers.  If  stu¬ 
dents'  loads  were  such  that  only  one  item  of  training  equipment 
were  required,  then  the  additional  costs  attributed  to  the 
actual  equipment  trainer  would  make  the  combination  more  costly 
and  no  more  effective  than  using  only  the  simulator  for  train¬ 
ing.  If  simulators  cost  less,  the  same  result  would  also  apply 
to  cases  where,  because  of  a  large  student  load,  two  or  more 
items  of  training  equipment  were  required.  The  school  might 
use  some  combination  of  simulators  and  actual  equipment  trainers 
This  type  of  compromise,  while  not  most  cost-effective,  might 
appear  reasonable  to  skeptics  who  believe  that  actual  equipment 
trar  ers  are  still  required.  Since,  except  for  Cicchinelli, 
Harmon,  Keller,  et  al. ,  1980,  there  are  no  data  to  support  or 
reject  the  notion  that  both  actual  equipment  trainers  and  simu- 
ators  are  needed  for  adequate  training,  there  is  much  to  be 
-ained  b>  collecting  the  job  performance  data  needed  to  resolve 
•_nis  dilemma. 

Maintenance  simulators,  it  has  been  argued,  cost  less  and 
are  more  effective  than  actual  equipment  trainers  because  they 
provide  feedback  to  students,  provide  training  in  a  larger  num¬ 
ber  of  malfunctions  than  is  otherwise  possible,  and  have  fewer 
breakdowns  when  used  by  students.  Cost  data  support  the  first 
claim;  although  the  other  claims  appear  plausible,  there  are  no 
data  to  support  (or  reject)  any  of  them.  Some  enterprising  mili 
tary  laboratory  is  invited  to  consider  these  questions. 

Maintenance  simulators  provide  individualized,  self-paced 
instruction  and,  because  of  this,  one  would  expect  them  to  save 
some  of  the  time  needed  by  students  to  complete  the  course  of 
instruction.  This  result  has,  in  fact,  been  reported  in  three 
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studies.  If  confirmed,  as  we  would  expect,  the  cost  avoidance 
attributable  to  reduced  expense  for  students'  pay  and  allowances 
at  school  would  increase  the  cost-advantages  of  simulators. 

This  type  of  calculation  has  not  been  included  in  any  evaluation 
of  maintenance  simulators. 

It  is  conceivable  that  some  maintenance  simulators  would 
be  more  expensive  to  procure  than  actual  equipment  trainers  for 
the  same  applications.  If  all  other  things,  e.g.,  effectiveness, 
are  equal,  then  we  should  obviously  choose  the  less  expensive 
option.  However,  "all  other  things"  are  rarely  equal.  A  simu¬ 
lator  though  more  expensive  to  procure  than  an  actual  equipment 
trainer,  might  sufficiently  shorten  student  tine  at  school,  re¬ 
ducing  the  need  for  instructors  and  support  personnel;  to  be 
less  expensive  on  a  life-cycle-cost  basis;  it  might  also  improve 
student  on-the-job  performance  sufficiently  to  be  cost-effective 
in  terms  of  the  combined  costs  of  training  and  (subsequent) 
maintenance.  This  statement  is  not  intended  to  be  an  argument 
in  favor  of  simulators.  Rather,  it  is  made  to  point  out  that, 
up  to  now,  ail  studies  of  simulators  and  actual  equipment 
trainers  have  been  one-point  comparisons,  i.e.,  equal  effec¬ 
tiveness  and  lower  costs  for  simulators.  Since  no  studies  have 
been  made  between  training  devices  of  differing  levels  of  both 
cost  and  effectiveness  and  that  extend  the  analysis  to  later 
performance  on  the  job,  it  is  not  yet  possible  to  look  for  an 
optimum  combination  of  maintenance  training  equipment. 

D.  RISK  OF  CONCURRENT  DEVELOPMENT 

There  is  an  obvious  advantage  in  the  concurrent  development 
of  operational  and  traininq  equipments  so  that  trainers  are 
available  when,  and  preferably  before,  the  operational  equipment 
is  first  fielded.  However,  this  practice  also  entails  risk, 
since  even  minor  changes  to  the  operational  equipment  may  result 
in  large  additional  costs  to  modify  the  training  simulators 
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(especially  in  the  areas  of  simulation  software  and  courseware). 
In  five  cases  that  have  been  identified  as  concurrent  develop- 
pments,  a  significant  portion  of  final  simulator  costs  has  been 
attributed  to  modifications  in  the  operational  equipment. 
Although  the  sample  is  small,  it  suggests  that  concurrency  will 
increase  the  costs  of  simulator  development  programs. 

It  also  follows  that  if  simulators  for  training  are  de¬ 
veloped  only  after  the  design  of  the  operational  equipment  has 
been  frozen,  the  simulators  may  not  be  ready  for  training  when 
needed.  A  possible  alternative  is  to  train  the  initial  cadre 
of  personnel  with  actual  equipment  or  with  simulators  based  on 
a  preliminary  design,  krowing  that  more  adequate  simulators 
will  be  built  later.  Whether  or  not  such  an  alternative  is 
both  effective  enough  and  not  too  costly  is  a  to  >ic  for  system¬ 
atic  study. 

E.  IMPORTANCE  OF  SOFTWARE  COSTS 

Software  and  courseware  (i.e.,  program  design  and  pro¬ 
gramming)  appear  to  be  major  elements  of  cost  in  non-standard 
maintenance  simulator  systems.  No  hard  data  were  found  on  this 
point;  nevertheless,  it  is  the  opinion  of  individuals  who  have 
been  involved  with  the  management  of  maintenance  simulator  pro¬ 
grams.  According  to  these  individuals,  cost  overruns  that  have 
occurred  have  been  due  primarily  to  problems  in  developing 
software  programs.  Should  this  be  true,  it  points  to  a  cost 
advantage  to  be  gained  by  developing  widely  applicable  soft¬ 
ware  systems  for  the  more  complex  training  applications. 

Although  no  data  are  available  on  this  point,  this  would  not 
apply  to  standard  systems,  since  the  same  software  system  is 
employed  in  all  applications  developed  by  one  contractor. 


F.  LIMITED  POTENTIAL  FOR  QUANTITY  PRODUCTION 

The  bulk  of  potential  maintenance  simulator  applications 
appears  to  arise  in  system-specific  training  (as  opposed  to 
general  skill  training),  and  a  majority  of  the  more  promising 
candidate  applications  seem  to  be  associated  with  aviation 
training.  However,  training  for  a  specific  model  of  aircraft 
is  concentrated  at  a  small  number  of  sites  and  involves  low 
rates  of  student  flow.  As  a  result,  there  is  a  limited  poten¬ 
tial  for  quantity  production  of  a  given  model  of  simulator 
over  which  development  costs  can  be  amortized  and  a  limited 
opportunity  for  reducing  unit  costs  through  a  widespread  adop¬ 
tion  of  maintenance  simulators. 

G.  WHAT  SHOULD  BE  DONE  NEXT 

1 .  Cost-Effectiveness  Trade-Off  Studies 

Maintenance  simulators  have  been  found  to  be  cost-effective, 
although  the  data  for  this  finding  are  limited.  There  is  no 
reason  to  doubt  the  same  result  for  additional  comparisons  of 
maintenance  simulators  and  actual  equipment  trainers.  Neverthe¬ 
less,  we  should  know  how  to  optimize  the  design  and  use  of  main¬ 
tenance  simulators  and  to  be  able  to  make  trade-offs  between 
their  effectiveness  and  cost.  There  is  almost  a  total  lack  of 
systematic  knowledge  about  the  relation  (i.e.,  trade-offs) 
between  effectiveness  and  cost  in  the  design  and  use  of  mainte¬ 
nance  simulators;  for  example,  what  features  increase  their 
effectiveness  in  particular  applications;  conversely,  little 
is  known  regarding  the  relationships  between  simulator  features 
and  their  costs.  Simulators  can,  naturally,  increase  in  cost 
in  many  ways,  such  as  by  including  more  malfunctions  in  their 
courseware  programs,  by  providing  more  complete  realism  in 
appearance  and  functional  capabilities  (in  both  3-D  and  2-D 
designs),  and  by  providing  more  computer-based,  instructional 
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guidance  to  students;  the  converse  of  any  of  these  statements 
may  also  be  considered.  A  substantive  question  is  to  deter¬ 
mine  the  extent  to  which  increases  in  the  capability  of  mainte¬ 
nance  simulators  (with  associated  increases  in  cost)  improve 
the  effectiveness  of  training,  i.e.,  student  performance, 
beyond  that  which  can  be  achieved  without  these  incremental 
costs.  No  studies  have  been  undertaken  to  explore  such  func¬ 
tional  relationships,  except  for  the  still-to-be-completed 
effort  of  Cicchinelli,  Harmon,  Keller,  et  al . ,  1980  that  will 
compare  2-D  and  3-D  versions  of  the  6883  Test  Stand. 

2.  Validate  Simulators  with  Performance  on  the  Job 

Student  performance  at  school  is,  at  best,  an  indirect 
measure  for  evaluating  the  benefits  of  simulators,  compared  to 
actual  equipment  trainers,  at  schools.  The  real  issue  is  to 
compare  how  training  with  either  of  these  devices  improves  the 
ability  of  course  graduates  to  maintain  equipment  on  the  job. 
The  purpose  of  school  training  is  to  qualify  students  to  per¬ 
form  well  on  jobs  in  the  field  and  not,  per  se,  to  complete 
a  course  at  school.  Data  to  show  the  effectiveness  of  main¬ 
tenance  simulators,  compared  to  actual  equipment  trainers,  as 
measured  by  field  performance,  is  totally  lacking  and  is  essen¬ 
tial  for  definitive  evaluations.  Cicchinelli,  Harmon,  Keller, 
et  al.,  1980,  an  oft-cited  reference  in  this  paper,  reported 
supervisors'  ratings  of  on-the-job  performance  of  course  grad¬ 
uates  but  did  not  collect  objective  data  on  the  actual  perfor¬ 
mance  of  these  individuals. 

3 .  Fidelity  of  Simulation 

Instructors,  in  general,  favor  the  use  of  actual  equip¬ 
ment,  rather  than  simulators,  for  the  training  of  maintenance 
personnel.  Reasons  given  for  this  preference  are  that  students 
need  to  train  with  actual  equipment  and  that  the  lack  of  real¬ 
ism  in  simulators  can  interfere  with  effective  training.  Such 
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reasons  cannot  be  dismissed,  because  the  views  of  instructors 
can  influence  the  way  in  which  simulators  are  used  in  a  course; 
an  inappropriate  use  of  a  simulator  may  easily  make  it  not  inef¬ 
fective  and  therefore  not  efficient  for  training. 

A  numLer  of  studies  have  shown  that  low-cost  devices,  such 
as  mock-ups,  charts,  and  flat  panel  simulators  are  as  effective 
as  reaJ  equipment  for  training  personnel  to  operate  (rather  than 
maintain)  equipment  (Grimsley  1969;  Denenberg  1954;  Torkelson 
1954;  Swanson  1954;  Vris  1955;  Spanqenberq  1974;  French  and 
Martin  1957;  Prophet  and  3oyd  1970,  Pouqherty,  Houston,  and 
Nicklas  1957;  and  Cox,  Wood,  Boren  et  al.,  1965.  Useful  reviews 
of  this  topic  may  be  found  in  Micheli  1972,  Kinkade  and  Wheaton 
1972,  and  Fink  and  Shriver  1978).  These  studies  show  that  stu¬ 
dent  achievement  (i.e.,  learninq  the  required  information)  is 
about  the  same  with  real  equipment,  expensive  simulators,  or 
inexpensive  mockups;  this  is  taken  to  represent  a  range  of  high 
to  low  fidelity  in  these  devices.  Some  studies  have  shown  that 
there  no  differences  between  individuals  trained  on  high  or  low 
fidelity  devices  when  measured  by  training  time,  amount  of  in¬ 
formation  remembered  (after  4  or  6  weeks),  or  time  devoted  to 
additional  training  some  time  after  leaving  school.  These 
findings  apply  primarily  to  teaching  procedural  tasks,  e.g., 
nomenclature,  equipment  start-up,  malfunction  location,  and 
troubleshooting  logic.  This  evidence  cannot  be  denied,  but  it 
has  not  had  a  major  influence  on  the  design  or  procurement  of 
maintenance  simulators. 

All  recent  studies  of  maintenance  simulators  have  evaluated 
a  specific  simulator  as  a  direct  alternative  to  some  actual 
equipment  for  training  purposes.  Fidelity  was  not  varied  sys- 
temically  or  otherwise  in  any  of  these  studies,  with  one  excep¬ 
tion.  Flat  panel  (2-dimensional)  and  3-dimensional  versions  of 
the  6883  Test  Station  simulator  were  developed  so  that  a  direct 
comparison  could  be  made  of  their  effectiveness  for  training 
maintenance  technicians.  The  three-dimensional  version,  produced 


103 


by  Honeywell,  has  been  evaluated  for  use  ir.  training  technicians 
to  operate  and  maintain  the  6883  Test  Station  (Cicchinell i , 
Harmon,  Keller,  et  al..  1980).  The  two-dimensional  version, 
produced  by  Burtek,  will  also  be  evaluated,  but  data  collection 
and  evaluation  had  not  been  completed  when  this  report  was 
written  (April  1981). 

There  has  been  no  effort  to  interpret  what  aspects  of  fi¬ 
delity,  if  any,  may  have  been  varied  in  the  studies  that  have 
been  completed  to  date,  although  it  is  obvious  that  different 
pieces  of  equipment  with  different  methods  of  presentation 
were  involved  and  that  thece  devices  cost  different  amounts  of 
money.  There  is,  at  present,  no  way  of  measuring,  scaling,  or 
defining  what  we  mean  by  the  fidelity  of  a  training  device, 
particularly  with  respect  to  its  effectiveness  for  training 
students.  A  distinction  made  by  Miller  (1954)  between  psycho¬ 
logical  simulation  and  engineering  simulation  (and  copied 
assiduously  ever  since)  does  not  help  very  much:  "engineering 
simulation  [isl  the  copying  of  some  physical  model  and  its 
physical  properties"  (p.  19;  emphasis  in  original);  "psycholog¬ 
ical  simulation....  provides  stimuli  so  that  responses  learned 
to  them  will  transfer  from  training  [with  training  devices!  to 
operations  with  little  or  no  loss"  (p.  19).  "Psychological 
simulation  may  be  far  removed  from  physical  realism"  (p.  20). 
"The  development  of  training  devices  should  rest  on  psychologi¬ 
cal  simulation  rather  than  engineering  simulation"  (p.  20). 

It  may  be  that  loss  expensive  devices  are  as  effective 
as  more  expensive  ones  for  maintenance  training.  However,  we 
lack  both  a  metric  and  a  guideline  to  identify  either  the 
physical  or  functional  characteristics  of  these  devices  that 
influence  the  effectiveness  of  training.  The  interrelation¬ 
ships  of  complexity,  fidelity,  and  cost  of  training  equipment 
and  the  transfer  of  training  from  training  devices  to  oper¬ 
ational  equipment  clearly  deserve  systematic  attention,  both 
for  R&D  on  training  devices  in  general  and  for  particular 
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emphasis  on  maintenance  training.  Different  levels  of  com¬ 
plexity  and  of  fidelity  may  be  required  for  manual,  hands-on 
skills  needed  in  maintenance  than  for  those  which  involve  pri¬ 
marily  knowledge  and  procedures  associated  with  diagnosis  of 
malfunctions  and  troubleshooting. 

4 •  On- the- Job  Training 

Technical  training  at  school  qualifies  a  maintenance  tech¬ 
nician  to  undertake  further  training  on  the  job  and  is  not 
expected,  by  itself,  to  produce  a  high  level  of  competence. 

At.  stake,  therefore,  is  assessing  the  cost-effective  mix  of 
training  at  school  and  on  tne  job.  This  important  question  is 
beyond  the  scope  of  this  paper.  Nevertheless,  a  potential 
advantage  of  maintenance  simulators,  particularly  as  the  newer 
ones  become  more  compact  and  portable,  is  that  they  would  per¬ 
mit  us  to  measure  the  performance  of  maintenance  personnel  on 
or  near  their  job  sites  and,  where  deficiencies  are  found,  to 
provide  refresher  training  to  particular  individuals.  Thus, 
maintenance  simulators  provide  a  means  of  collecting  objective 
data  about  technicians  on  the  job  (in  a  test-like  situation), 
that  could  be  used  to  validate  not  only  the  use  of  simulators 
in  school  but  of  actual  equipment  trainers?  this  also  applies 
to  any  other  feature  of  interest  in  the  type  of  instruction 
offered  at  school. 

5 .  R e s e a rch  on  Ma i n tenance 

Even  after  about  30  years  of  research  on  maintenance 
training,  we  still  lack  some  fundamental  information  about  how 
humans  perform  the  task  of  maintenance.  As  a  consequence,  we 
cannot  accurately  specify,  as  suggested  above,  whether  a  partic¬ 
ular  simulator  should  be  simple  or  complex,  two-  or  three- 
-dimensional  in  form,  the  optimum  mix  of  general  and  specific 
maintenance  training,  and  the  trade-offs  between  increased  re¬ 
liance  on  automatic  and  built-in  test  equipment  versus  reliance 
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on  human  ability  to  diagnose  and  perform  various  maintenance 
procedures.  At  least  in  principle,  it  is  feasible  to  improve 
built-in  test  equipment  to  assist  the  technician  to  find  mal¬ 
functions  and  to  design  a  system  so  that  components  and  test 
points  are  more  accessible  to  maintenance  personnel.  The  real 
issue  is  to  determine  whether  increased  expenditures  during 
system  development  for  engineering  characteristics  to  facilitate 
maintenance  will  reduce  expenditures  for  personnel,  training, 
maintenance,  test  equipment,  and  spare  parts  over  the  life 
cycle  of  that  system. 

It  is  not  yet  possible  to  measure  the  complexity  of  main¬ 
tenance  tasks  so  that  specifications  for  equipment  which  have 
an  impact  on  maintenance  and  maintenance  personnel  can  be  set 
both  for  the  design  of  weapons  systems  and  for  maintenance 
simulators  and  training  programs  (see  Wohl  1980?  Rouse,  Rouse, 
Hunt,  et  al.,  1980;  Nauta  and  Bragg  1980). 

It  is  not  yet  clear  to  what  extent  maintenance  simulators 
should  be  desiqned  to  provide  generic  training  applicable  to  a 
variety  of  equipments  and/or  specific  training  applicable  primar¬ 
ily  to  particular  models  of  equipment.  A  current  program  at  the 
Navy  Personnel  Research  and  Development  Center  is  attempting  to 
address  this  issue  (the  Generalized  Maintenance  Trainer  System). 

There  are  insufficient  data  on  the  amount  of  time  re¬ 
quired  to  find,  identify,  and  fix  various  types  of  malfunctions. 
Without  such  data,  there  appears  to  be  little  rational  basis  for 
determining  the  priority  to  be  given  to  various  types  of  mainte¬ 
nance  tasks  included  in  maintenance  training  courses  and,  of 
course,  in  the  design  of  the  maintenance  simulators  to  be  used 
in  these  courses  (Johnson  and  Reel  1973).  The  work  of  Rouse, 
Rouse,  Hunt,  et  al.  (  198 CM  suggests  that  the  more  difficult 
fault  isolation  tasks  are  in  equipment  with  feedback  loops; 
humans  benefit  durinq  training  when  they  are  given  immediate 
knowledge  of  results  about  the  rules  they  are  using  to  identify 
faults;  these  skills  appear  to  be  transferable  to  situations 
where  immediate  knowledge  of  results  is  not  provided. 
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APPENDIX  A 


SUMMARY  OF  STUDIES  EVALUATING  TRAINING  WITH 
MAINTENANCE  SIMULATORS  AND  ACTUAL  EQUIPMENT  TRAINERS 


SIMULATOR 


Generalized  Sonar 
Maintenance  Trainer 
(GSMT) 


COURSE 


COURSE 

LENGTH 

(STANDARD) 


,,,  ...  Sonar  maintenance 

'  '  '  '  (Special  course  for 

this  experiment) 


4  days 


AET: 


Fleet  Sonar  School 
San  Dieqo,  CA 


SIM: 


EQUIPMENT 


AN/SQ-4  Sonar 


GSMT 


TYPE  OF 
INSTRUCTION 


AET 


Simulator 


NO.  OF 
STUDENTS 


SCOR 


Accuracy: 
Same  or  he 


Note:  Final  test  for 
both  groups  on 
AN/SQS-23  sonar 


Criterion 

Testi^) 


Intermediate  general 
electronics  (4-week 
segment  of  14-week 
course  in  Sonar  Main¬ 
tenance  Training) 

Fleet  Sonar  School 
Key  West,  FL 


4  weeks 


AET 


SIM:  GSMT 


ALT  1 
AET  2 


(2) 


20 

20 


54.4 

54.9 


Simulator  20 


54.6 


^Trains  calibration  alignment,  preventive 
maintenance  and  troubleshooting  of  cir¬ 
cuits  and  components  common  to  six 
sonar  systems. 

( 2 ) 

'Control  group  1  trained  before  and  control 
group  2  trained  after  experimental  gi  up. 

^Percent  correct  answers  to  special  test 
with  141  items. 

(4) 

This  is  a  low-power  sonar  system  rather 
than  a  true  simulator. 


COMPARISON 


Average  22  percent  Students  favorable 
faster  in  perform¬ 
ing  maintenance 
tasks  on  test 

y: 

hotter 


nn  Avg.  of 
weekly 
tests _ 

85.  6 
H  ?  ? 


COMMENTS 


Transfer  of  training 
experiment.  Students 
trained  on  AET  or  simu¬ 
lator;  both  groups 
tested  on  a  new  sonar. 
Performance  compared  on 
five  maintenance  tasks. 
Simulator  group  super¬ 
ior  but  differences  not 
statistically  signifi¬ 
cant. 

No  significant  differ¬ 
ence  between  groups. 
Analysis  of  data  shows 
that  although  students 
trained  with  GSMT  had 
lower  "academic  poten¬ 
tial"  (GCT/ARI  scores) 
than  control  groups, 
they  performed  as  well 
as  controls.  Report 
does  not  describe  equip¬ 
ment  used  to  train 
control  groups. 


REFERENCES 


Parker  and  DePauli, 
1967 


DePauli  and  Parker, 
1969 


A- 1/A- 2 


SIMULATOR 


COURSE 


EQUIPMENT 


COURSE 

LENGTH 

(STANDARD) 


TYPE  OF 

INSTRUCTION!  STUDENTS 


APQ-1ZG  radar  for  AD  7 
aircraft 

Air  National  Guard 
Buckley  field,  CO 


Mohawk  Propeller  System 
( t>J  C51 ) ,  OV-1  Airframe 
Repair  Course 
U.S.  Army  Transporta¬ 
tion  School ,  Fort 
Lustls,  V A 


Hydraulic  and  flight 
control  system,  T-ZO 
aircraft 


Lnglne,  power  plants 
and  fuel  system 


Z4  hrs 


Lnvi ronment/uti  1 1  ty 
system 


Mock-ups  and 
breadboards ; 
conventional 
classroom 
Instruction 


Arresting  gear  and 
speed  brake 
trainer,  main  and 
aux'llary  landing 
gear  trainer, 
fl Ight  control 
trainer. 

NAS  Chase  Field 

Elevator  and  ele¬ 
vator  trim  panel , 
aileron  and  trim  panel 
hydraulic  speed  brake 
panel ,  landing  gear 
panel  rudder  and 
rudder  trim  panel , 
wheel  brakes  and 
flaps  panel. 

NA'  Meridian 


AET :  Fuel  systems  train¬ 
er;  engine 
NAS  Chase  Field 

SIM:  Fuel  panel; 

DC  electrical  start 
and  run  panel 
NAS  Meridian 


AET  :  Heat  and  vent 
training  unit; 
seat , 

NAS  Chase  Field 


' 1  ^Qn- the- job  training;  length  not  specified, 

(Zhl  qualified,  6  untrained. 

^40  items,  multiple  choice  . 

(41 

'Oral  performance  test  on  T-ZC  aircraft  at  end  of  course;  aircraft  part  Identification, 
knowledge  of  maintenance  manual;  situational  troubleshooting.  Scored  by  examiner  from 
Naval  Weapons  Engineering  Support  Activity. 


Perform 


Pet  torn 


Wright  and  Campuel 1  , 
1975 


SIMULATOR 


FC- 1 1  (continued) 


Lt-IIVLP' 


COURSE 

COURSE 

LENGTH 

(STANOARD) 

EQUIPMENT 

Weapon  control  system 
mechanic ,  Block  VI , 
radar 

Lowry  AFP,  CO 


Motor  Transport  School 
Marine  Curps  Base 
Camp  Lejeune,  NC 


FM  lot  familiarization 
for  T-2C  aircraft 
NAS  Pensacola,  FL 


Naval  FI Ight  Officer 
familiarization  for 
TA-4C  aircraft, 

NAS  Pensacola  ,  FI. 


60  hrs 


1H  hrs' 


SIM:  environmental 

simulation  panel , 
environmental  and 
utility  panel ; 
seat , 

NAS  Meridian 


ALT:  AN/AF-Q  126 

radar  for  A-70 
a  1  rc  ra  f  t 


SIM:  LC-II 


AET:  mobile  tralnlny 
unit, 

chalkboard 
SIM:  IC-IIVI.P 


(as  above) 


TYPE  CF  NO.  OF 
INSTRUCTION  STUDENTS 


S 1  :nu  1  j  tor 


j  mu  1  a  tor' 


ATT 

Si  mu  I  a  tor 


7 

Performance 

End  of,3) 

course 

i 

Performance 

24 

Normal  opera¬ 
tions  cheeky 
(95  Items); 
solving  fi  mu' 
function  pro: 

IS 

Normal  opera, 
tlons  checkoi 
j  (N. S) ;  sol v  1 1 
j  malfunction 
problems  (N/ 

(:  pilots 

30  NLOV  5 

Large  panel  version,  Intended  for  class  demonstrations. 

^Light  (B)  lesson  units,  e.y,,  electrical  systems,  instruments,  ejection  system. 


'Naval  Flight  Officers. 


I 


COMPARISON 

SCORES 

TIME 

attitudes  towards 

SIMULATORS 

COMMENTS 

Hbt-fcHbNUtS 

( 4 ) 

f’or  forma  nee v 

B4.5 

l  rui  ofl3)  35.4 

i.ourse 

Students  lave, die; 
i  nstrui  tors  n  n !  ral  tn 
nega live 

1  erf  ormance^ 

76.4 

Normal  opera¬ 
tions  checkout 
(46  items); 

‘■olvinq  8  mal  - 
function  problems 

Students  neutral  to 
favorable;  instructors 
favor  siniulati  r.  3  cau- 
t  ious 

No  significant 
difference  between 

ACT  and  simulator 
group 

McGuirk,  Pieper  and 

Miller,  1975 
(Also  Miller  and 

Rockway,  1975) 

Normal  ouera- 
t i uns  checkout 

1  IN  S),  solving 
mal funct ion 
problems  (N.S) 

Cost  estimate  for 
equipment  (2  sets)  in 
complete  course: 

$1 ,068,000  ALT  vs 
$169,000  for  simulators 

No  data  in  paper,  EC-11 
judged  effective  for 
training  and  recom¬ 
mended  for  adoption, 

Project  savings  of 
$386,000  over  15  years 

Platt.  1976 

Student  s  ami  instructors 
moderately  t.e  h iijhly 
favorable  t.n  \|M  ([C-I I ) 

finding  based  on  fac¬ 
tor  analysis  of  atti¬ 
tudes 

Biersner  1975 

I ns t  rue t I .ivor  siwu- 
1  tit.or  ovi’t*  train- 

Ifuj  ,iif|’, 

Author  Judges  training 
with  simulator  to  be 
equal ly  effective  to 
use  of  AET  a t  school ; 
performance  on-the-job 
unknown 

Biersnrr  1976 

i 

'as  .i'hivp) 

(as  above) 

Biersner  1976 

a 
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i 


COMPARISON 


r 

SCORES 

TIME 

ATTITUDES  TOWARDS 
SIMULATORS 

COMMENTS 

J _ 

Provides  CA1  and  CMI 
services  to  multiple 
student  stations;  one 
Instructor  station 

No  evaluation  reported 

Evaluated  favorably  by 

3  Navy  Instructors  for 
fidelity  of  simulation 
and  usefulness  for 
training 

As  above,  plus  com¬ 
ments  by  attendees  at 
demonstrations  that 
"AEMT  approach  ap¬ 
peared  to  provide  as 
good  if  not  better 
training  effective¬ 
ness  than  is  achieved 
using  operational  hard¬ 
ware,"  (p.27).  No  per¬ 
formance  data 

No  evaluation  reported 

No  evaluation  reported 

REFERENCES  j 


Modrlck,  Kanarlck, 
Daniel  and  Gardner, 
1975 


Modrlck,  Kanarlck, 
Daniel ,  and  Gardner, 
1975 


Daniels,  Datta, 
Gardner,  and  Modrlck, 
1975 


Modrlck,  Kanarlck, 
Daniel ,  and  Gardner, 
1975 


Modrlck,  Kanarlck, 
Daniel ,  and  Gardner, 
1975 


A-7/A 


COURSE 

SIMULATOR 

COURSE 

LENGTH 

(STANDARD) 

EQUIPMENT 

TYPE  OF 
INSTRUCTION 

NO.  OF 
STUDENTS 

SCORE 

General  i zed  Maintenance 
Training  System  ( GMT S ) - 
Rigney  System 

AN/ SRC - 20  UHF  Voice 
Communications  System 
Advanced  Electronics 
Schools,  Department 
Facility,  Naval  Schools 
Command,  San  Diego,  CA 

AeT 

Simulator 

20 

AN/SPA-66  radar 
repeater , 

Naval  Mobile  Technical 

Unit  S,  San  Diego,  CA 

4  days 
(16  nrs) 

AFT 

Simulator 

10 

AN/WSC-3  trat  jceiver 
for  fleet  satell  i  te 
communication  system. 
Advanced  Electronic 

School  Division 

Service  School  Command, 

San  Diego,  CA 

I 

I 


COMPARISON 


SIMULATOR 


COURSE 

LENGTH 

(STANDARD) 


EQUIPMENT 


TYPE  OF 
INSTRUCTION 


NO.  OF 
STUDENTS 


Fault  Identification 
Simulator  (FIS) 


Hagen  Automatic  Roller 
Control 

Fleet  Tralnlnq  Center 
San  IMcgo,  CA 


'Pneumatic 
Maintenance  Slmu- 
1  a  to  r \ and  Boiler 
Control  replica 

FIS  and  Boiler 
Control  replica 


Simulator 


'Actual  equipment  trainer 

^Actual  components  activated  by  pneumatic  and  electrical  signals 
individualized,  self-paced  Instruction  compared  to  conventional  Instru.  Mon  above 


EQUIPMENT 


6883  Converter/ 
Flight  Control 
Systems  Test 
Station 


3-dlmenslonal  simu¬ 
lation  of  a  hove 


TYPE  OF 
INSTRUCTION 


Control 


,i  mu  Citin' 


NO.  OF 
STUDENTS 


59 


SCORES 


Test* 


1) 


Assist 

needed 

Instru 


56 


23. 1 

13.9 

23. 9 
Same 


22.8 

14.0 

23.0 

Same 


COMPARISON 


SCORES 


Assistance 

1  < ,  >  needed  by 

Is'  '  Instructor^ 


21,1 
I J .  q 
V>3. 9 
jSarnc 


?,H 
4 ,  l) 
1.0 
.uni’ 


?,? 


?.  i 


TIME 


Tocomplete  tost 


54,?  ni(n 


54 , 1!  in  t  n 


ATTITUDES  TOWARDS 
SIMULATORS 


5 1 iicli’ii t  \  i.ivni’.iHe; 

I  nst  nn  I  in',  neutral  to 
S  ]  H|lif  I  y  I  ivmMl’Ie 


COMMENTS 


This  Test  Station  used 
only  two  days  In  regu¬ 
lar  23  weeks  course 
(6  days  In  this  test) . 
Test  shows  equal  effec¬ 
tiveness  at  school  and 
In  follow-up  on  job  2-32 
weeks  later,  based  on 
supervisors'  comments. 
Study  also  says  simula¬ 
tor  costs  about  one- 
third  that  of  actual 
equipment  to  acquire 
and  use. 


REFERENCES 


Clcclnel 1 1 ,  Harmon, 
Keller  and  Kottenstette 
(1980) 


A-13/A-14 


a:  *4  '  -  ' 


SIMULATOR 


Paper  and  pencil 
training  aids 


Trai  ner-Tpster 


Punchboard  Tutnr 


P’ape r  and  pend' 
training  aids 

Trainer-Tester 


Custom-built 
■  linulator6' 


Flow  Diagram  Trainer  and 
Automated  Microfiche 
Terminal 


COURSE 

COURSE 

LENGTH 

(STANDARD) 

Basic  electronics 
U.S.  Naval  Training 
Center,  Great  Lakes, 
IL 


Radar  repair 


Army  Signal  School, 
Fort  Monmouth,  NJ 


JDA  radar  display 
HMS  Collingwood 


EQUIPMENT 


10  wks  AET:  (1  ) 

AET  +  Trainer-Tester 

AET  +  Punchhoard  Tutor 

9  wks  Taped  lectures(N=4) 

Trainer-Tester 

Custom-built  simulator 


2  days  Panel  board  simulator 

Microfiche  projector 

Equipment  mock-up 
(JDA  radar  simulator) 


^Push-pull,  three-stage  transmitter  superheterodyne  receiver,  twelve  4S-m1nute  classes  for  each 
(2) 

'  '50  mul tipi 2-cholce  Items 

^Grade  assigned  by  instructor 
(4i 

lb  multiple-choice  Items 
( b) 

'Simulator  malfunctions,  tests  and  measurements  on  specially  prepared  paper  layouts  of  equipment 
developed  by  Van  Valkenburgh,  Nooger,  and  Neville,  Inc.,  1954 

^Locally  deslyned  to  be  more  realistic  than  Trainer-Tester;  uses  schematic  drawings 


TYPE  OF  NO.  OF 
INSTRUCTION  STUDENTS 


Troubl 

shoot 

Exam 

Lab 

Troubl 

shoot 


6  groups 


N=26 ,  27  each 


No.  ot 
to  fine 
fault  hr i 


COMMENTS 


REFERENCES 


COMPARISON 


SCORES 


1"  m 

>at1nn^4^,  1 


I  .1 

m 


ible- 
)oti  mj 


TIME 


ATTITUDES  TOWARDS 
SIMULATORS 


Students  In- 1  if vp  aids 
i  nip  roved  t  n  ml  .In '.hooting 
skills. 


I nstructurs  pn  f'  r  lab 
work  to  1  i'ii i ner  Tester; 
least  accept  f’um  hhoard 
Tester 


No  significant  differ¬ 
ence  between  groups 
trained  differently 
when  tested  later  In 
Advanced  training 

Comounlcatlons  ( N“ 1 26 ) ; 
Trainer-Tester  group 
(N'210)  superior  on 
laboratory  grades 


Cantor  and  Brown,  1956 


II  1  u 

|ih  1 e- 

boting  65 


ot  checks 
ind  faults 
before  after 


Average  time  to 
find  faults  (min) 
rault  before  after 


111  A 

lb  5 

5 


1  1U  5 

2  21  6 

3  --  6 


Effects  measured  at  the 
end  of  the  course  by 
performance  test  (find 
malfunction  in  actual 
radar  components)  and 
written  tests 


Glass,  1967 


Main  finding  Is  that 
lectures  on  troubleshooting 
Improve  effectiveness 
of  paper  simulators. 
Custom-built  simulators 
are  more  effective  than 
Trainer-Tester 


Training  with  panel  board 
trainer  and  microfiche 
projector;  before  and 
after  tests  with  JDA 
radar  simulator;  no 
comparison  with  AET 


Cunningham,  1977 


SIMULATOR 


COURSE 


Computer  simulations  for 
training  In  fault  diagnosis 
(computer-assisted  instruc¬ 
tion) 


1 1 1 


COURSE 

LENGTH 

(STANDARD) 


EQUIPMENT 


Aircraft  power  plant  condi 
tionlng  and  testing 

Institute  of  Aviation 
'.In  1  vers  1  ty  of  111  Inols 


3  special 
training 
sessions 
(total  of 
six  hours) 
In  semester 
course 


Context-free  simulation 

Context-specific  simulation 

Instructional  TV  film  on 
troubleshooting  engines 


Continental  0-300  recipro¬ 
cating  (on  test  stand) 


TYPE  HO.  OF 

INSTRUCTION  STUDENTS 


Perform-  Pe 
ance  up, 
score  at' 


Instruction¬ 
al  TV 


ac. 

4.4 


Context-free  12 
fault  diag¬ 
nosis 


4.0 


Lycoming  0-23b  reciprocating  Context-  12  4,n 

engine  (on  test  stand)  specific 

fault  diag¬ 
nosis 


fault-finding  performances  on  two 
aircraft  engines  on  test  stands. 


r 

. 

COMPARISON 

SCORES  ( 1 1 

TIME01 

ATTITUDES  TOWARDS 
SIMULATORS 

COMMENTS 

REFERENCES 

Perform-  Percent  of  Percent  of 

ance  approprl-  Inappro- 

si  ore  ate  prlate 

actions  actions 

4.4  44".  0.9* 

4.0  L’3  0.7 

4.0  30  1.4 

1 

\ 

l 

1 

} 

[ 

1 

T  line/ 
Problem 

1 . 3  hrs 

1 .8 

Att  itmlr'  ('>  i’“>st  favorable) 

■1  . 

■l.-l 

Transfer  measured  from 
method  of  training  to 
fault  finding,  5  mal¬ 
functions,  In  2  actual 
engines  on  test  stand. 
Training  with  Instruc¬ 
tional  TV  yielded  best 
troubleshooting  perform¬ 
ance,  judged  due  to 
similarity  of  training 
of  training  with  test. 
Evidence  that  computer 
simulation  transfer  also 
found  but  less  pronounced. 

Johnson,  1980 

■  j  A-17/A-1L 

./ 
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APPENDIX  B 


PROGRAM  COSTS  OF  NON-STANDARD  SIMULATORS 

A.  SUMMARY 

Table  B-l  provides  a  summary  of  costs  for  nine  non-standard 
simulator  programs  for  which  data  were  available.  Table  B-2 
shows  the  "normalized"  costs  of  these  programs.  Normalization 
involved  two  adjustments  to  the  program  costs. 

(1)  In  order  to  provide  a  consistent  base  for  comparison 
among  programs,  recurring  production  costs  were  di¬ 
vided  by  the  quantity  fabricated.  Thus,  each  program 
will  reflect  the  costs  that  (hypothetically)  would 
have  been  incurred  if  only  one  trainer  had  been  pro¬ 
duced  . 

(2)  Two  of  the  research  programs  (the  6883  and  MA- 3  Test 
Benches)  incurred  significant  costs  for  evaluations. 
These  costs  were  excluded  so  that  each  program  will 
reflect  the  costs  that  (hypothetically)  would  have 
been  incurred  if  they  had  been  intended  for 
main-line  training. 

The  values  shown  in  Table  B-2  provide  the  basis  of  the  percent 
ranges  of  the  different  cost  elements  shown  in  Table  15. 

The  remainder  of  this  Appendix  provides  a  short  discussion 
of  each  of  the  nine  programs  and  a  more  detailed  display  of 
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B. 


RESEARCH  PROGRAMS 


The  three  research  programs  (simulating  the  Navy  A-7E  HUD 
and  MA-3  and  the  Air  Force  6883  Test  Benches)  have  had  mixed 
histories.  Each  was  conceived  as  a  vehicle  for  an  extensive 
evaluation,  comparing  the  effectiveness  of  maintenance  simu¬ 
lation  as  an  alternative  to  actual  equipment  for  training, 
and  there  is  no  evidence  that  they  were  intended  for  use  in 
regular  training  programs.  However,  current  planning  envisions 
using  the  MA-3  Test  Bench  for  training.  These  three  devices 
are  the  only  ones,  to  date,  that  address  simulation  for  inter¬ 
mediate  (as  distinct  from  organizational)  echelon  maintenance 
training. 

Each  of  the  trainers  is  a  unique  3-dimensional  device, 
and  each  has  had  a  different  principal  contractor.  For  the 
two  Navy  devices,  the  contractors  had  no  prior  experience  in 
developing  similar  systems.  The  contractor  for  the  6883  (Honey¬ 
well)  previously  had  built  one  model  of  a  3-dimensional  mainte¬ 
nance  simulator. 

1 .  Honeywell  6883  Test  Bench 

The  6883  device  was  procured  as  part  of  a  continuing  main¬ 
tenance  simulation  research  project  of  the  Technical  Training 
Branch  of  the  Air  Force  Human  Resources  Laboratory  (AFHRL). 

It  is  the  only  one  of  the  three  research  devices  that  has  been 
extensively  evaluated;  results  have  been  published  recently  in 
Deignan  and  Cicchinelli  (1980)  and  Cicchinelli,  Harmon,  Keller, 
and  Kottenstette  (1980).  This  evaluation  addressed  the  simulator 
for  training  operations  and  maintenance  of  the  6883  Test  Bench. 

A  standardized  2-D  system,  also  simulating  the  6883,  has  recently 
been  delivered  and  will  be  evaluated  for  training  operation  and 
maintenance  of  the  test  bench. 

Program  cost  information  (Table  B- 3 ) ,  based  on  contract 
actuals  and  in-house  costs,  was  obtained  from  the  AFHRL  program 
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TABLE  B-3 . 


6883  TEST  BENCH  (Dollars,  Thousands) 


Denver 

Contractor 

Ir, -House 

Research 

Honeywell 

Institute 

Engineering 

Total 

Managamant 

Training 

Original 

Changes 

•  Program 

Task/Function 

and 

Evaluation 

Contract 

and 

Support 

Maintenance 

Contract 

Non-Recurring  Cost 

' 

Front  End  Analysis 

19 

f 

19 

Task  Analysis 

Other 

(19 

(19 

Oaslgn  and  Development 

26 

313 

88 

427 

Hardware 

34 

1 

35 

Software/ Courseware 

26 

273 

85 

384 

Tachnlcal  Data 

Othar 

6 

2 

8 

Hardwara  Fabrication 

(non-racurrlng) 

57  . 

57 

Tast  and  Evaluation 

1S5 

155 

Program  Managamant 

14b 

145 

Total  Non-Recurring 

171 

155 

389 

88 

803 

Racurrlno  Cost 

Production 

169 

169 

Hardwara  Fabrication 

(recurrlna)1 

Special  Tools/Test 

149 

149 

Equipment 

Initial  Spares 

Othar 

20 

20 

Logistic  Support 

104 

104 

Interim  Maintenance 

104 

104 

Support 

Othar 

Initial  Training 

Total  Recurring 

169 

104 

273 

Program  Total  Cost 

171 

155 

55B 

192 

1,076 

*  Including  Installation  and  chackout. 
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office.  Honeywell  contracted  for  this  program  at  roughly  the 
same  time  as  its  VTAR  system  was  delivered  to  the  Navy.  There 
appears  to  be  little  similarity  between  the  two,  and  it  is 
unlikely  that  the  prior  VTAS  experience  proved  significant  in 
the  6883  program. 


2.  A- 7 F  HUD  Test  Bench 


The  A-7E  HUD  ( He ad-Up- Di splay )  Test  Bench  is  one  part  of 
a  Navy  research  proqram  that  envisioned  development  and  evalua¬ 
tion  of  six  3-dimensional  maintenance  simulators  --  one  each 
for  training  in  three  types  of  skills  (electronic,  electro¬ 
mechanical,  and  mechanical)  at  two  maintenance  echelons  (organ! 
zational  and  intermediate).  In  addition  to  evaluation  of  these 
different  applications  of  simulators,  the  research  proqram  was 
to  result  in  developing  procedures  for  formulating  simulator 
development  specifications. 

The  A-  7E  HUD  is  the  intermediate  echelon,  electronic 
skills  portion  of  this  research  program.  To  date,  the  device 
has  not  been  evaluated.  An  evaluation  program  was  desiqned 
and  the  simulator  was  delivered  to  the  training  site.  However, 
the  program  was  not  initiated,  and  its  future  is  in  doubt. 

This  was  the  first  device  to  be  initiated  under  the 
research  program,  and  the  first  of  its  kind  for  the  Navy. 
Personnel  involved  with  management  of  the  program  (the  Human 
Factors  Laboratory  of  the  Naval  Training  Equipment  Center)  are 
frank  to  state  that  it  was  a  learninq  experience  for  them  in 
terms  of  simulator  specification  (one  objective  of  the  research 
program)  and  development  rrocedures.  The  device  was  initially 
limited  to  simulation  of  the  "day"  version  of  the  HUD  that 
was  a  mature  A-7E  system  at  the  time.  However,  a  FLIR  (forward 
looking-infrared)  version  was  in  development  at  that  time,  and 
the  simulator  program  was  later  expanded  to  incorporate  the 
FLIP  capability.  This  opened  the  program  to  two  problems  -- 
modification  and  concurrency. 
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Program  costs  (shown  in  Table  B-4)  were  developed  by  the 
Human  Factors  Laboratory  from  contract  records  and  the  expertise 
of  Laboratory  personnel.  The  costs  show  the  impact  of  the 
decision  to  simulate  the  FLTR  capability.  Six  contractors  were 
involved  in  the  program,  three  of  these  (including  the  A-7E 
prime  contractor)  because  ot  the  FLIR  decision.  Over  one-third 
of  the  program  total  costs  (and  over  40  percent  of  contracted 
costs)  resulted  from  the  FLIR  mod i L icat ion .  Unfortunately, 
there  is  no  way  to  separate  FLI R-associated  costs  into  those 
arising  from  modification,  per  se ,  and  those  arising  from  con¬ 
current  development  of  the  simulation  and  the  operational 
equipment.  Program  office  personnel  believe  the  latter  was  a 
significant  element,  and  state  that  durinq  several  intervals 
development  of  the  simulator  was  ahead  of  development  and  docu¬ 
mentation  of  the  operational,  equipment. 

1.  MA- I  Test  Bench 

The  MA- 3  is  a  second  element  of  the  Navy  maintenance  simu¬ 
lator  research  program,  in  this  case  addressing  training  of 
intermediate  echelon,  electro-mechanical  skills.  This  program 
was  initiated  more  than  two  years  after  the  A-7E  HUD  and  bene¬ 
fited  significantly  from  the  earlier  experience,  according  to 
program  office  personnel.  The  device  was  delivered  durinq  the 
summer  of  1980  and  is  currently  undergoing  evaluation  that 
should  be  complete  in  June  1981. 

Program  costs  are  shown  in  Table  R-5.  These  data  were 
developed  from  contract  records  and  expertise  of  personnel 
from  the  Human  Factors  laboratory  that  managed  the  program. 

The  simulator  has  one  noteworthy  characteristic,  differ¬ 
ent  from  any  of  the  other  non-standard  devices,  that  is  com¬ 
pletely  unrelated  to  its  role  as  a  research  vehicle.  It  is 
the  only  device  of  its  class  to  provide  training  that  is  not 
specific  to  a  sinqle  weapon  or  support  system.  The  MA- J  is  a 
universal  stand  used  throughout  the  Navy  for  on-shore  testing 


TABLE  B-4.  A-7E  HUD  TEST  BENCH  ( 


Contractor 

In-Houso 

Hcneywel 1 

Educational  Computer 
Corporation 

Vought 

Aircraft 

Appllmation, 

Tnc. 

AACTS 

Engineering 

Inc. 

Data 

General 

| 

L 

Initial 

Day  System; 

Day  System; 

Day  System; 

FL IR  System, 

F(.IR  System, 

Total 

Program  1 

Design 

Front  End 

Original 

Engineering 

Expansion; 

Expansion; 

FUR  System, 

Maintenance 

Taik/Functlon 

anil 

Support 

Analysis 

Contract 

Change 

Data  Base 
Design 

Design 

Expansion; 

Hardware 

Contract 

f 

Non-Recurring  Cost 

Front  End  Analysis 

4.1 

95 

20 

30 

45 

234 

Task  Analysis 

20 

65 

20 

30 

45 

180 

Other 

24 

30 

_54 _ 

Design  and  Development 

r: 

215 

48 

115 

IDS 

S 

650 

Hardware 

15 

20 

15 

25 

5 

80 

Software/Courseware 

4R 

185 

28 

100 

160 

521 

Technical  Data 

H 

15 

10 

33 

Other 

It- 

16 

Hardware  Fabrication 

(non-recurring) 

21 

24 

Test  and  Evaluation 

Program  Management 

/;' 

72 

Total  Non-Recurring 

212 

95 

235 

48 

145 

240 

.  5.  _ 

980 

Recurring  Cost 

Production 

95 

60 

20 

30 

20 

• 

225 

Hardware  Fabrication 
(recurring)® 

Special  Tools/Test 

90 

45 

20 

25 

20 

200 

Equipment 

5 

Initial  Spares 

Other 

5 

15 

20 

Logistic  Suoport 

/6 

76 

Interim  Maintenance 

1 

Support 

76 

76 

Other 

Initial  Training 

5 

5 

5 

15 

Total  Recurring 

100 

65 

20 

30 

25 

76 

316 

Program  Total  Cost 

21? 

95 

335 

113 

165 

270 

30 

76 

1,296 

!_  1 

Including  installation  and  checkout. 
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TABLE  B-5.  MA-3  TEST  BENCH  (Dollars,  Thousands) 


Contractor 


Task/Function 


Non-Recurrinq  Cost 


Front  End  .Analysis 
Task  Analysis 
Other 


Design  and  Development 
Hardware 

Software/Courseware 
Technical  Data 
Other 


Hardware  Fabrication 
(non-recurring) 

Test  and  Evaluation 

Program  Management 


Total  Non-Recurring 


nq  Cost 


Production 

Hardware  Fabrication 
(recurring)® 
Special  Tools/Test 
Equipment 
Initial  Spares 
nther 


Logistic  Support 
Interim  Maintenance 
Support 
Other 


Initial  Training 
Total  Recurring 


Program  Total  Cost 


Appllmatlon,  Inc. 


In-House 

Engineering 

and 

Support 


Seville 

Training 

Cost/ 

Effective- 


Total 

Program 


Including  Installation  and  checkout. 
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of  all  generators  and  constant-speed-drives  that  comprise  air¬ 
craft  primary  electric  power  systems.  (A  related  test  stand, 
the  MA-2,  serves  the  same  function  for  all  primary  aircraft 
power  systems  on  board  ship.)  As  a  result,  knowledge  of  both 
its  operation  and  maintenance  is  a  widespread  requirement  and, 
if  exploited,  a  variant  of  this  simulator  (modified  for  main¬ 
line  training)  might  satisfy  an  extensive  training  requirement. 

The  MA-3  consists  of  two  distinct  components  --  the  test 
stand  itself  and  associated  equipment  (such  as  electronic  test 
sets  and  oil  coolers)  to  adapt  its  use  to  the  rarge  of  different 
generators  and  constant  speed  drives  that  are  used  on  various 
Navy  aircraft.  The  simulator  was  designed  with  this  ir  mind 
and  maintains  a  separation  of  these  two  components.  Although 
the  current  simulator  was  built  specifically  for  training  AV-8A 
maintenance  personnel,  the  contractor  has  provided  the  following 
rough  estimates. 

•  Modification  of  the  current  simulator  system  to  allow 
simulation  of  other  generator/ constant-speed-drive 
combinations  would  cost  $15,000  to  $20,000  (including 
computer  programming  and  fabrication)  for  each  combina- 
t  ion . 

•  Follow-on  units  in  lots  of  two  of  the  current  configura¬ 
tion  (including  hardware,  software,  and  documentation) 
would  cost  $170,000  per  unit. 

Two  points  should  be  noted  with  regard  to  the  second  con¬ 
tractor  estimate.  The  first  is  that  the  current  simulator  is 
configured  for  research  and  it  can  be  anticipated  that  quantity 
production  of  a  device  that  did  not  contain  features  needed  for 
research  would  cost  significantly  less.  The  second  point  is 
that  there  is  an  apparent  inconsistency  between  the  contractor's 
estimate  and  the  costs  developed  by  Human  Factors  Laboratory 
personnel.  The  latter  costs  attribute  only  $99  thousand  to  re¬ 
curring  device  production  and  $145  thousand  to  recurring  program 
costs  (and  include  all  non-recurring  fabrication  costs  that,  in 


in  fact,  were  incurred).  We  have  no  explanation  for  the  differ¬ 
ence  in  estimates.  However,  the  lower  Laboratory  estimate  is 
consistent  with  data  obtained  on  other  programs.  Either  esti¬ 
mate  totals  to  much  less  than  the  cost  incurred  in  the  research 
program,  and  the  bulk  of  this  cost  reduction  can  be  attributed 
to  the  high  proportion  of  non-recurring  costs  associated  with 
one-of-a-kind  production. 

C.  A- 6  TRAM  DRS 

The  A- 6  TRAM  DRS  (Detection  and  Ranging  System  -  AN/AAS-33A) 
simulator  was  initiated  as  a  research  program,  i.e.,  as  one  ele¬ 
ment  of  the  Navy  research  program  that  includes  the  A- 7  HUD  and 
the  M A- 3  Test  Stands.  However,  this  aspect  of  the  program  was 
modified  at  an  early  stage,  and  the  simulator  was  adopted  as 
one  of  the  primary  devices  for  organizational  echelon  mainte¬ 
nance  training  of  the  electronic  portion  of  the  DRS.  Two  units 
were  built  and  shipped  to  the  two  A-6  training  detachments 
where  they  were  placed  in  main-line  training  after  a  relatively 
short  checkout  and  acceptance  test  period. 

This  appears  to  be  one  of  the  more  successful  non-standard 
simulator  programs.  The  devices  cannot  be  considered  high  cost 
and  have  been  well  received  by  training  personnel.  However, 
certain  features  of  the  device  and  its  procurement  point  up 
problems  that  seem  to  recur  with  maintenance  training  simulators. 
Four  of  these  are  discussed  below. 

•  The  simulator  provides  only  for  organizational  mainte¬ 
nance  training  on  the  electronic  portion  of  the  DRS. 

As  a  result,  it  did  not  relieve  a  requirement  for  using 
operational  equipment  for  training  on  the  mechanical 
portion  of  the  DRS.  While  the  existence  of  the  simula¬ 
tor  reduces  the  training  load  placed  on  the  operational 
equipment,  it  does  not  necessarily  reduce  the  amount 
or  cost  of  actual  equipment  required  for  training. 
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•  This  model  of  DRS  is  peculiar  to  the  A-6  aircraft.  A-6 
training  is  conducted  at  only  two  locations,  and  the 
two  devices  serve  to  fully  satisfy  the  training  require¬ 
ment.  This  is  typical  of  aircraft  systems  today,  even 
those  that  are  procured  in  large  numbers.  When  it  is 
considered  that  the  major  portion  of  simulator  program 
cost  is  nonrecurring  in  nature  (approximately  75  per¬ 
cent  in  this  case)  there  appears  to  be  little  promise 

in  roducinq  their  costs,  except  by  use  of  standardized 
systems  where  high  development  costs  (such  as  for  the 
software  systems)  can  be  spread  to  a  number  of  training 
applications . 

•  This  is  the  first  device  of  this  type  to  be  contracted 
for  by  Grumman,  and  its  capabilities  and  design  (includ¬ 
ing  the  software  package)  are  highly  tailored  to  this 
single  training  application.  Configuration  changes  and 
modifications  are  common  to  current  combat  aircraft  and 
even  apparently  minor  ones  may  result  in  major  changes 
to  maintenance  simulators  whose  costs  may  be  a  major 
portion  of  those  required  for  developing  a  new  device. 

•  Decisions  regarding  the  timinq  of  training  device  deliv¬ 
eries  are  critical  in  providing  training  on  new  or  modi¬ 
fied  operational  equipment.  Whatever  types  of  training 
devices  are  used,  they  must  be  in  place  before  that 
training  can  commence,  and  training  must  be  provided  to 
personnel  before  the  operational  equipment  can  become 

an  effective  part  of  the  force.  Since  simulation 
trainers  require  their  own  development,  this  must  occur 
concurrently  with  development  of  the  operational  equip¬ 
ment.  However,  the  operational  equipment  is  subject  to 
frequent  modification  during  development  and  for  a  con¬ 
siderable  period  after  its  fielding.  As  discussed  above, 
these  modifications  can  have  a  drastic  impact  on  simula¬ 
tor  costs.  An  examnle  of  this  impact  can  be  seen  in 
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the  A-6  DRS  program.  The  DRS  tactical  configuration 
and  the  simulator  were  developed  at  the  same  time,  both 
by  Grumman,  and  the  DRS  configuration  was  not  finalized 
at  the  time  the  simulator  program  was  initiated.  The 
simulator  contract  provided  an  allowance  for  changes 
in  the  simulator  that  were  the  result  of  anticipated 
modifications  to  the  operational  equipment,  and  at  the 
time  of  final  delivery  the  cost  of  these  changes  amounted 
to  35  percent  of  the  final  Grumman  contract  value. 

Program  cost  information  (Table  B-6)  was  obtained  from  the 
program  procurement  manager  at  the  Naval  Training  Equipment  Cen¬ 
ter.  At  present,  both  trainers  have  been  delivered  and  placed 
in  service;  the  costs  shown  in  Table  B-6  appear  to  be  the  total 
costs  to  the  government,  except  for  updating  the  devices  to  re¬ 
flect  recent  modifications  to  the  operational  equipment. 

D.  VTAS 

The  VTAS  simulator  had  its  origin  in  a  change  in  configu¬ 
ration  of  the  F-4J  aircraft  that  entailed  changes  in  training 
equipment  then  employed  at  two  Naval  Air  Stations  (NAS).  An 
assessment  of  alternati ves  for  these  changes  concluded  that 
simulation  would  cost  between  60  and  85  percent  of  the  alter¬ 
natives  that  involved  modifying  or  procuring  additional  opera¬ 
tional  equipment  for  training. 

The  contractor  (Honeywell)  has  produced  several  non-standard 
maintenance  simulators.  However,  VTAS  was  the  first  and  appears 
to  be  quite  different  from  Honeywell's  later  programs.  The 
relative  distribution  of  program  costs  among  cost  elements  is 
quite  different  from  that  on  other  programs  (including  the 
later  Honeywell  programs) .  It  is  concerned  only  with  a  single 
system  that  is  not  complex  i elative  to  current  avionics  systems. 
The  Navy  procurement  program  manager  described  the  training 
requirement  as  relatively  simple,  and  the  simulator  reflects 
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TA3LE  B-6.  A-6  TRAM  DRS  (Dollars,  Thousands) 


Contractor 


Task/Function 


Non-Recurrinq  Cost 


Front  End  Analysis 
Tasr  Analysis 
Other 


Design  and  Development 
Hardware 

Software/Courseware 
Technical  Data 
Other 


Hardware  Fabrication 
(non-recurring) 

Test  and  Evaluation 

Program  Management 

Total  Non-Recurring 


Recurring  Cost 


Production 

Hardware  Fabrication 
(recurring)® 
Special  Tools/Test 
Equipment 
Initial  Spares 
Other 


Logistic  Support 
Interim  Maintenance 
Support 
Other 


Initial  Training 
Total  Recurring 


Program  Total  Cost 


Including  Installation  and  checkout. 
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the  limitation  (providing  eight  malfunctions  that  are  integrated 
into  the  contractor-proprietary  software  system). 

Cost  information  (Table  B-V)  was  obtained  from  the  con¬ 
tractor.  In  addition  to  the  contract  costs,  the  procurement 
program  manager  has  estimated  that  between  cne-half  and  three- 
quarters  of  a  man-year  was  expended  by  in-house  civilian  and 
military  personnel  on  various  functions  (including  front-end 
analysis  and  program  management). 

E.  AT-TRAINER 

The  AT-Trainer  takes  its  name  from  its  purpose  of  provid¬ 
ing  training  in  all  equipments  maintained  by  the  Navy  aviation 
technician  (AT)  rating  on  three  series  of  the  F-4  aircraft-- 
the  communicar.ior/navigation/identif ication  and  the  electronic 
countermeasures  suites.  It  was  bequn  as  a  small  scale  in-house 
project  at  the  North  Island  Naval  Air  Rework  Facility  to  pro¬ 
vide  simulation  of  one  model  of  L’T.F  communication  equipment  on 
the  F-4N .  Since  that  time,  it  has  expanded  through  a  series 
of  program  changes  to  encompass  all  AT-maintained  equipment  on 
the  F-4N .  F-4J,  and  RF-4B  and  the  inertial  navigation  equipment 
of  the  RF-4B  maintained  by  the  aviation  electrician  (AE)  rating. 

This  simulator  provides  only  for  organizational  mainte¬ 
nance  training  that  is  cypicallv  confined  co  troubleshooting 
of  installed  equipment.  This  involves  a  large  number  of 
equipments  the  controls  and  indicators  of  which  are  located  on 
the  aircraft  instrument  panel:  the  physical  configuration  of 
the  simulator  constitutes  an  extensive  mock-up  of  the  cockpits. 
Tne  particular  equipments  simulated,  by  aircraft  series,  are 
shown  in  Table  B-8. 

At  program  completion  three  units  will  have  been  built. 

Two  of  these  (to  be  used  at  Beaufort  MCAS  and  Oceana  NAS)  will 
be  limited  to  providing  AT  training  for  the  F-4J.  The  third 
(delivered  to  El  Toro  MCAS)  will  provide  simulation  for  AT 
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TABLE  B-7.  VTAS  (Dollars,  Thousands) 


Contractor 

Honeywel  1 

Task/Function 

Total 

Program 

Non-Recurrlnq  Cost 

Front  End  Analysis 

Task  Analysis 

Other 

Design  and  Development 

74 

Hardware 

19 

Software/Courseware 

38 

Technical  Data 

17 

Other 

Hardware  Fabrication 

(non-recurring) 

Test  and  Evaluation 

6 

Program  Management 

51 

Total  Non-Recurring 

131 

Recurrlnq  Cost 

Production 

163 

Hardware  Fabrication 

(recurring)® 

156 

Special  Tools/Test 

Equipment 

Initial  Spares 

7 

Other 

Logistic  Support 

Interim  Maintenance 

Support 

Other 

Initial  Training 

3 

Total  Recurring 

166 

Program  Total  Cost 

297 

Including  Installation  *nd  checkout. 
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TABLE  B-8.  SIMULATED  AT-TRAINER  EQUIPMENT,  BY  AIRCRAFT  SERIES 


Equipment  Designation  (Function) 

Aircraft  Series 

F-4N 

RF-4B 

F-4J/S 

Communlcation/Navlqatlon/  Identification  Equipment 

AN/ASQ-19 

(Integrated  Electronics  Central) 

X 

AN/ARC-159 

(OHF  Communications) 

X 

X 

AN/ ASQ- 1 60 

(Integrated  Electronics  Central) 

X 

AN/ARC-105 

(UHF  Communications) 

X 

AN/ ARN - 1 1 8 

(TACAN ) 

X 

AN/AJB-3 

(Computer) 

X 

AN/AJB-7 

(Computer) 

X 

AN/ASN-39 

(INS) 

X 

X 

AN/ASN-59 

(Attitude,  Heading,  and  Reference) 

V 

A 

AN/ASN-92 

(Carrier  Alignment  JNS) 

X 

AN/ASW-25A 

(Data  Link) 

X 

AN/ASW-25B 

(Data  Link) 

X 

AN/ASM-23 

(use) 

X 

X 

X 

ECM  Equipment 

AN/ALR-45 

(ECM) 

X 

X 

AN/ALR-50 

(ECM) 

X 

X 

AN/ALQ-1 26 

(ECM) 

X 

X 

AN/ALE-29 

(Chaff  Dispenser) 

X 

AN/ALE-39 

(Chaff  Dispenser) 

X 

X 

AN/ALM-70 

GSE  (AN/ALE-29) 

X 

AN/ALM-164 

GSE  (AN/ALE-39) 

X 

X 

AN/ASM-456 

GSE  (AN/ALR-45) 

X 

X 

AN/ALM-140  GSE  (AN/ALR-50) 

X 

X 
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training  in  all  three  aircraft  series  in  addition  to  AE  training. 
The  multiple  simulation  capability  is  provided  by  "quick  change" 
kits  to  reconfigure  the  simulator  to  each  aircraft. 

The  El  Toro  unit  serves  as  a  test  bed  for  the  program.  It 
was  delivered  as  a  partially  completed  device  in  mid-1979.  Fur¬ 
ther  capabilities  (in  terms  of  the  equipments  simulated)  are 
delivered  as  they  are  developed,  and  the  system  will  not  be 
completed  until  mid-1981.  The  Oceana  and  Beaufort  units  were 
delivered  in  December  1980  without  ECM  simulations;  this  capa¬ 
bility  will  be  installr  )  during  198  1  as  a  field  modification. 

Throughout  its  term,  the  AT-Trainer  proqram  has  been 
carried  on  strictly  as  an  in-house  activity.  The  tasks  involved 
include  development  of  simulation  hardware  (including  some  of 
the  data  processor)  and  the  software  operating  system.  As  a 
result,  it  is  a  unique  system.  Apparently,  there  are  no  plans 
to  carry  these  developments  any  further  or  to  employ  them  in 
other  simulations. 

The  AT  program  costs  are  shown  in  Table  B-9.  The  program 
manager  feels  these  values  may  understate  true  costs  of  the 
project,  because  many  of  the  individuals  involved  provided 
extensive  unpaid  time.  In  addition,  the  costs  are  probably 
not  comparable  to  those  for  other  simulator  programs.  The 
accounting  methods  employed  will  differ  from  those  used  by 
contractors  so  that  various  categories  of  costs  may  have  dif¬ 
ferent  meanings. 

F.  INTEGRATED  RADIO  ROOM 

This  is  one  of  two  programs  to  incorporate  simulation  into 
the  initial  design  of  the  training  proqram  for  a  new  major 
weapon  system.  A  total  of  four  simulators  are  employed  in  the 
complete  Trident  training  system.  Two  are  associated  with  the 
Integrated  Radio  Room  (the  communications  system)  while  the 
other  two  are  associated  with  pneumatics  and  are  standardized 
simulation  systems. 


TABLE  B-9 .  AT  TRAINER  (Dollars,  Thousands) 


Contractor 

North  Island  Naval  Air  Research 
Facility 

Task/Function 

Initial 

Modlf- 

mm 

Program 

cations/ 

Updates 

Non-Recurring  Cost 

Front  End  Analysis 

Task  Analysis 

Oth«r 

20 

20 

20 

Design  and  Development 

920 

95 

1 ,015 

Hardware 

355 

50 

405 

Software/Courseware 

415 

45 

460 

Technical  Data 

Other 

150 

150 

Hardware  Fabrication 

(non-recurring) 

267D 

g 

276 

Test  and  Evaluation 

8 

8 

Program  Management 

40 

40 

Total  Non-Recurring 

1,255 

104 

1,359 

Production 

Hardware  Fabrication 

440 

11 

451 

(recurring)* 

Special  Tools/Test 
Equipment 

410 

11 

421 

Initial  Spares 

Other 

30 

30 

Logistic  Support 

Interim  Maintenance 
Support 

Other 

Initial  Training 

30 

30 

Total  Recurring 

470 

11 

481 

Program  Total  Cost 

1 ,725 

.. 

115 

1 ,840 

*  Including  Installation 

and  checkout. 

Includes  $205  for  depot  repair  facility. 
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Trident  communications  system  personnel  are  responsible 
for  both  operation  and  maintenance  of  the  system;  and  both 
functions  are  trained  in  the  same  course  employing  the  same 
training  equipment.  Three  major  training  devices  are  used, 
designated  A,  B,  and  C;  two  of  these  devices  (A  and  B)  provide 
both  operation  and  maintenance  training,  while  trainer  C  is 
devoted  to  maintenance  training. 

•  Trainer  A  is  a  reconstruction  of  the  on-board  radio 
room.  It  consists  of  a  complete  set  of  the  operational 
equipment  (along  with  training-unique  equipments)  to 
provide  operator  team  and  watch-standing  training  and 
hands-on,  on-equipment  maintenance  training. 

•  Trainer  B  is  a  reconstruction  of  a  part  of  the  radio 
room  using  only  simulation  equipment.  Its  function  is 
to  provide  individual  and  team  operations  training  and 
training  in  the  system  fault  isolation  and  diagnostic 
capabilities  of  the  operators'  console,  and  the  use  of 
the  built-in  test  equipment 

•  Trainer  C  consists  of  both  simulated  and  tactical 
equipment.  It  is  a  series  of  part-task-trainers  to 
provide  training  in  troubleshooting  and  fault  isolation 
procedures  at  a  module/component  level  and  hands-on 
preventive  and  corrective  maintenance. 

This  program  provides  a  definitive  demonstration  of  both 
complementarity  and  substitutability  between  simulation  and 
operational  equipment.  Two  trainers  are  used  for  operator 
training,  and  all  three  for  system  maintenance  training.  Early 
studies  of  Trident  training  requirements  identified  two  alterna¬ 
tives  for  radio  room  training.  One  alternative  was  the  current 
program  combining  operational  equipment  and  simulators.  The 
second  alternative  proposed  only  operational  equipment  trainers 
(consisting  of  three  of  the  current  A- Trainers). 

RCA  is  the  contractor  for  the  operational  equipment  as 
well  as  the  three  trainers;  both  the  trainers  and  the  operational 
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« 
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equipment  were  developed  concurrently.  The  training  equipment 
contract  is  RCA’s  first  experience  in  developing  simulators. 

One  could  expect  the  simulation  trainers  to  be  closely  patterned 
after  the  operational  equipment  and,  perhaps,  to  be  quite  dif¬ 
ferent  from  other  non-standard  simulators. 

Our  opinion  that  these  simulators  differ  from  others  is 
bolstered  by  comparing  the  relative  cost  of  software/courseware 
in  this  program  with  other  programs.  Roughly  15  percent  of  RCA 
costs  are  attributed  to  software/courseware,  while  the  average 
for  the  other  non-standard  systems  is  close  to  35  percent.  The 
operational  equipment  employs  extensive  data  processing,  and  the 
relatively  low  software/courseware  costs  for  the  simulators 
would  be  consistent  with  RCA  incorporating  its  detailed  knowl¬ 
edge  of  the  operational  equipment  software  into  the  development 
of  the  simulator  software  system--rather  than  developing  a 
wholly  new  system,  as  appears  to  be  the  case  of  other  non¬ 
standard  simulators. 

Simulator  program  costs  (shown  in  Table  B-10)  were  obtained 
from  the  program  office  for  Trident  training  (at  the  Naval 
Training  Equipment  Center) .  They  are  based  on  the  original 
cost  proposal  and  the  program  changes  that  have  been  negotiated 
to  date. 

G.  F- 16 

The  F-16  maintenance  simulation  system  is  the  most  ambi¬ 
tious  non-standard  program  undertaken  to  date.  Like  the  Trident 
Radio  Room,  the  use  of  simulation  was  incorporated  into  the 
initial  design  of  the  maintenance  training  program  and  the 
simulators  were  developed  concurrently  with  the  operational 
equipment.  Unlike  Trident,  though,  simulation  was  developed  by 
a  second  contractor  (Honeywell)  under  subcontract  to  the  weapon 
contractor  (General  Dynamics). 
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TABLE  B - 1 0 .  TRIDENT  INTEGRATED  RADIO  ROOM  (Dollars,  Thousands) 


- - 

American 

Contractor 

RCA 

ECC 

Systems 

In-House 

'nrpnrat.inn 

Uperator/Malntenance 

Total 

Trainer 

Maintenance  Trainer  | 

Task 

Program 

and 

Program 

Task/Function 

Basic 

Contract 

Basic 

Contract 

Analysis 

Office 

Management 

Contract 

Changes 

"retract 

Changes 

Support 

Support 

Non-Recurrlnq  Cost 

Front  End  Analysis 

400 

270 

6/0 

Task  Analysis 

Other 

400 

270 

670 

Design  and  Development 

1 ,856 

307 

1  ,417 

27 

3,607 

Hardware 

1 ,238 

204 

886 

20 

2,348 

Software/lourseware 

431 

89 

353 

6 

879 

Technical  Data 

Other 

187 

14 

178 

1 

380 

Hardware  Fabrication 

(non-recurring) 

Test  and  Evaluation 

12 

1? 

24 

Program  Management 

210 

65 

145 

9 

248 

140 

817 

Total  Non-Recurring 

2,078 

372 

1  ,5/4 

36 

400 

248 

410 

5,118 

Recurring  Cost 

' 

Production 

1  ,124 

58 

506 

2 

1 ,690 

Hardware  Fabrication 

1,103 

(recurring)® 

725 

58 

318 

2 

Special  Tools/Test 

1 1 7 

Equipment 

78 

39 

470 

Initial  Spares 

Other 

321 

149 

-  -  - 

Logistic  Support 

47 

4/ 

94 

Interim  Maintenance 

94 

Support 

Other 

47 

4/ 

- -  --  - 

. . . .  *  ’ 

Initial  Training 

114 

4 

68 

186 

Total  Recurring 

1,285 

62 

621 

2 

_ 

_ 

1,970 

Program  Total  Cost 

3,363 

434 

2.195 

J _ 

38 

400 

248 

410 

7,088 

e  Including  installation  and  checkout. 
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The  simulators  encompass  eight  of  the  aircraft  systems, 
and  are  to  be  used  by  both  USAF  and  NATO  country  personnel. 

The  simulators  are  configured  as  both  flat  panels  and  cockpit 
mock-ups.  The  quantities  procured,  by  aircraft  system  simulated, 
are  shown  in  Table  B-ll.  Typically,  the  simulation  of  one  air¬ 
craft  system  employs  r.wo  panels  or  one  panel  and  a  cockpit 
mock-up . 


TABLE  B-ll.  F- 1 6  SIMULATOR  QUANTITIES  PROCURED,  BY  AIRCRAFT  SYSTEM 


Aircraft  System 

Number  oi 
Per  S 

;  Elements 
imulator 

Number  of  Simulators 
Ordered 

Panels 

USAF 

NATO 

Environmental  Control 

2 

3 

2 

Navigation 

2 

3 

2 

Fire  Control 

1 

i 

3 

3 

Flight  Control 

2 

3 

3 

Hydraulic 

2 

3 

3 

Electrical 

2 

3 

2 

Weapon  Control 

2 

3 

3 

Engine  Start 

1 

3 

3 

Engine  Diagnosis 

2 

3 

3 

Engine  Operation 

i 

3 

2 

USAF,  in  buying  three  sets  of  the  simulator  system,  will 
receive  48  panels  (of  16  different  types),  six  cockpit  mock-ups, 
and  30  processors;  NATO  countries,  in  total,  are  procuring  two 
or  three  trainers  for  each  aircraft  system. 
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The  F-16  maintenance  simulator  family  resembles  a  stand¬ 
ardized  system.  A  common  model  of  processor  and  a  common  soft¬ 
ware  system  is  used  for  all  training  applications  (panels/mock- 
ups).  The  F-16  system,  along  with  the  AWACS  simulators  dis¬ 
cussed  below,  may  well  be  the  first  members  of  a  new  standard¬ 
ized  simulator  system.  One  result  of  these  simulator  programs 
is  Honeywell's  development  of  what  it  has  termed  the  "Data  Base 
Generator."  This  is  a  programming  language  intended  to  allow 
for  a  simple  manual  translation  of  technical  manual  information 
into  code  and  the  machine  translation  of  that  code  into  FORTRAN. 

The  F-16  program  provides  a  dramatic  example  of  the  prob¬ 
lems  that  may  arise  when  operational  equipment  and  training 
simulators  are  developed  concurrently.  The  initial  contract, 
in  September  1977,  provided  for  a  target  price  of  $7.5  million 
for  development  and  procurement.  Delivery  to  the  Air  Force  of 
the  first  set  of  18  panels  and  mock-ups  and  10  processors  was 
stipulated  for  September  1978. 

The  simulator  design  freeze  was  set  for  January  1978,  and 
the  configuration  was  to  be  based  on  the  production  version  of 
the  aircraft.  However,  the  technical  documentation  of  the  pro¬ 
duction  aircraft  configuration  was  not  available  at  that  date; 
the  aircraft  continued  to  undergo  engineering  changes,  and  a 
lag  developed  in  documentation  of  the  changes.  The  aircraft 
configuration  changes  imposed  configuration  changes  and  rework 
of  the  simulation  models  (panel  elements,  software,  and  course¬ 
ware);  the  lag  in  documentation  increased  the  amount  of  rework 
required  to  accommodate  the  changes. 

A  recent  contract  amendment  reset  the  target  price  of  the 
training  equipment  at  $28.9  million.  It  is  impossible  to  at¬ 
tribute  a  specific  portion  of  the  increase  to  the  problems 
associated  with  concurrency.  However,  it  appears  that  a  siz¬ 
able  portion  did  arise  from  this  cause. 
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Documentation  of  current  program  costs  includes  separate 
estimates  of  the  costs  (in  terms  of  the  target)  for  the  train¬ 
ing  devices  associated  with  each  aircraft  system  simulated  for 
each  of  the  seven  sets  of  simulators  currently  on-contract. 
These  estimates  are  shown  in  Table  B-12.  Note  that  the  total 
of  non-recurring  costs  are  attributed  to  simulator  set  number 
seven  and  that  the  values  shown  exhaust  the  contract  total 
target  cost.  (That  is,  all  costs  have  been  allocated  to  the 
individual  simulations  even  though  some  fraction  of  them, 
especially  of  the  non-recurring  costs,  are  support  functions 
that  are  truly  common  to  all  elements  of  the  program.) 

The  estimates  contained  in  Table  B-12  allowed  a  separation 
of  recurring  and  non-recurring  costs  for  the  simulations  asso¬ 
ciated  with  each  aircraft  system  shown  in  Table  B-13.  (Note 
that  the  method  used  for  separation  provided  a  slightly  dif¬ 
ferent  estimate  of  total  program  cost.)  The  resulting  ratios 
between  recurring  and  non  recurring  costs  are  quite  consistent 
with  other  non-standard  simulator  programs  (see  Figure  4). 

H.  AWACS  NAVIGATION 

The  Air  Force  plans  to  provide  training  simulators  for 
three  AWACS  systems--navigation,  radar,  and  possibly  the  data 
display/processor  systems.  The  navigation  system  has  been 
delivered  and  is  in  operation,  while  the  radar  system  trainer 
has  recently  been  placed  under  contract.  Both  these  simulators 
are  contracted  to  Honeywell.  Both  are  flat  panel  devices  and, 
considering  the  continuity  they  afford  Honeywell,  it  would 
appear  that  they  will  be  quite  similar.  However,  costs  of  the 
devices  will  be  significantly  different;  one  explanation  for 
the  large  difference  in  cost  is  that  the  naviation  trainer 
nas  a  single  student  station  while  the  radar  trainer  will 
have  10. 


TABLE  B - 1 2 .  F-16  SIMULATOR  COSTS  BY  AIRCRAFT  SYSTEM  AND  SIMULATOR  SET 

(Thousands  of  Dollars) 


TABLE  B - 1 3 .  F-16  SIMULATOR,  RECURRING  AND  NON-RECURRING  COSTS 

(Thousands  of  Dollars) 


Aircraft  System  Simulated 

r 

Total 

Cost 

Non- 

Recurring 

Cost 

Recurring 

Cost 

Cumulative 

Average 

Recurring 

Cost 

Fire  Control 

2756 

1640 

1116 

186 

Flight  Control 

4413 

2174 

2239 

373 

Navigation 

2956 

1699 

1257 

251 

El ectrical 

2621 

1269 

1352 

270 

Environmental  Control 

1872 

1110 

762 

152 

Hydraul ic 

1849 

852 

997 

166 

Weapons  Control 

2781 

1560 

1221 

203 

Engine  Start 

2440 

1125 

1315 

219 

Engine  Diagnostic 

4635 

2307 

2328 

388 

Engi ne  Operati ng 

2713 

1227 

1486 

297 

Total 

29,036 

14,963 

14,073 

The  navigation  system  simulator  is  the  first  major  train¬ 
ing  device  USAF  has  procured  for  this  system.  Prior  to  its 
delivery,  training  was  limited  to  providing  introductory  train¬ 
ing,  without  the  benefit  of  training  equipment,  at  Keesler  AFB 
and  transferring  graduates  to  an  AWACS  operational  base  for  on- 
the-job  training.  The  operational  base  represented  the  first 
hands-on  experience  received  by  the  students. 

Considering  the  continuity  (and  overlap)  in  the  AWACS  and 
F-16  programs,  all  three  training  systems  should  have  extensive 
similarities  in  important  features.  Both  the  AWACS  navigation 
trainer  and  the  F-16  systems  employ  Honeywell's  Data  Base 
Generator,  implying  similarities  in  the  software  systems  (with 
which  the  generator  must  be  compatable).  It  is  hard  to  escape 
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a  conclusion  that  these  three  programs  have  provided  Honeywell 
with  the  essentials  of  an  advanced  standardized  simulation  sys¬ 
tem  that  will  find  application  in  other  training  areas. 

Cost  information  on  this  program  was  provided  by  the  AWACS 
Project  Office  at  the  Electronics  System  Division  (Table  B-14). 
The  original  FPIF  contract  with  Honeywell  provided  for  a  target 
cost  of  $1274  thousand  and  a  ceiling  price  of  $1528.  One  en¬ 
gineering  change  was  negotiated  for  $60  thousand,  bringing  the 
total  cost  to  the  Government  to  $1588  thousand. 

The  contract  has  incurred  a  significant  cost  overrun. 
Honeywell  has  reported  costs  of  roughly  $200  thousand  over  the 
ceiling  price,  but  the  Project  C,cice  believes  the  total  over¬ 
run  is  approximately  $600  thousand  (or  40  percent  of  the  ini¬ 
tial  ceiling  price).  The  Project  Officer  attributes  the  un¬ 
reported  $400  thousand  overrun  to  Honeywell's  cost  in  develop¬ 
ing  its  Data  Base  Generator  (i.e.,  software).  (See  the  discus¬ 
sion  of  the  F-16  program,  above.)  Development  of  the  Data  Base 
Generator  was  an  independent  Honeywell  decision,  and  its  cost 
appears  to  have  been  charged  to  both  the  F-16  and  AWACS  pro¬ 
grams.)  The  program  costs  shown  in  Table  B-14  encompass  the 
Project  Office  estimates  of  total  costs  (both  reported  and  un¬ 
reported).  This  is  the  only  simulator  program  treated  in  this 
fashion.  Maintenance  of  the  simulator  is  provided  through  a 
separate  FFP  contract  between  Honeywell  and  the  Air  Force 
Logistics  Command  ( AFLC ) .  An  estimate  of  Project  Office  in- 
house  costs  has  not  been  obtained. 
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AACS 

ABC 

ACTS 

ADP 

AET 

AEMT 

AFHRL 

AFPTRC 

AFSC 
AIDE 
A  IS 
AMES 


AMSAS 

AMTE 

AMTESS 

AN 

APL 

ARI 

AT 

ATACS 

ATE 

AWACS 


ABBREVIATIONS 

Army  Area  Communications  Systems 
Automatic  Boiler  Control 

Adaptive  Computerized  Training  System  ( Perceptronics ) 
Automated  Data  Processing 
Actual  Equipment  Trainer 

Automated  Electronics  Maintenance  Trainer  (Honeywell) 
Air  Force  Human  Resources  Laboratory  (Brooks  AFB) 

Air  Force  Personnel  and  Training  Research  Center 
(now  AFHRL) 

Air  Force  Specialty  Code 

Automated  Instruction,  Direction  and  Exercise 
Avionics  Intermediate  Shop 

Aircraft  Maintenance  Effectiveness  Simulation 
(a  model  developed  by  XYZYX  Information 
Corporation  for  NTEC ) 

Advanced  Manpower  Concepts  for  Sea-Based  Aviation 
Systems 

Automated  Maintenance  Test  Equipment 

Army  Maintenance  Training  and  Evaluation  Simulation 

System  ( AR i /PMTRADE) 

Army-Navy 

Authorization  Parts  List 
Army  Research  Institute 
Action  Taken 

Army  Tactical  Communications  System 

Automatic  Test  Equipment 

Airborne  Warning  and  Control  System 


C-l 


BIT 

BITE 

CAI 

CAM-T 

CASEE 

CB 

CIWS 

CMI 

CNR 

CNTT 

CPFF 

CPIF 

CRS 

CRT 

CSD 

CUT 

DCS 

DRS 

DS 

DSB 

ECC 

ECM 

EEMT 

EPICS 

EIC 

ETM 

FCS 


Built-in  Test 
Built-in  Test  Equipment 

Computer-assisted  Instruction 

Consolidated  Aircraft  Maintenance  Training  (Air  Force 
hands-on  training) 

Comprehensiveness  Aircraft  Support  Effectiveness 

Evaluation 

Component  Breakdown 

Close-In  Weapon  System  (Phalanx  Gun  System) 

Computer-Managed  Instruction 

Communication  Navigation  Radar 

Chief  of  Naval  Technical  Training 

Cost  Plus  Fixed  Fee 

Cost  Plus  Incentive  Fee 

Component  Repair  Squadron 

Cathode  Ray  Tube 

Constant  Speed  Drive 

Cross  Utilization  Trained  (AF) 

Deputy  Chief  of  Staff 
Detection  and  Ranging  System 
Direct  Support 
Defense  Science  Board 

Educational  Computer  Corporation 
Electronic  Countermeasures 
Electronic  Equipment  Maintenance  Training 
(Navy  Class  A  Training  School,  Honeywell) 

Enlisted  Personnel  Individualized  Career  System 
Equipment  Identification  Code 
Extension  Training  Materials 

Fire  Control  System 
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FFP 

FIS 

FLIR 

FOMM 

FPIF 

FPJPA 

FRAMP 

FTD 

FVS 

GMTS 

GNS 

GS 

GSE 

HHC 

HUD 

ICAP 

I  HOMS 

I  level 

IMA 

IMTS 

INS 

IPSA 

IRR 

ISD 

I TDT 


JCN 

JPA 

JTPT 


Firm  Fixed  Price 

Fault  Identification  Simulator  (Navy) 
Forward-Looking  Infrared 

Functionally  Oriented  Maintenance  Manual 
Fixed  Price  Incentive  Fee 
Fully  Procedural i zed  Job  Performance  Aid 
Fleet  Readiness  Aviation  Maintenance  Personnel 
Field  Training  Detachment  (Air  Force) 

Fighting  Vehicle  System 

Generalized  Maintenance  Training  System  (Navy) 
Guidance  and  Navigation  System 
General  Support 
Ground  Support  Equipment 

Headquarters/Headquarters  Company 
Heads-up  Display 

Improved  Capability 

Intermediate  Hands-on  Maintenance  Simulators  ( NTEC ) 

Intermediate  Level  Maintenance 

Intermediate  Maintenance  Activity 

Integrated  Maintenance  Training  System  (NTEC) 

Inertial  Navigation  System 

Integrated  Personnel  Systems  Approach 

Integrated  Radio  Room 

Instructional  System  Development 

Integrated  Technical  Documentation  and  Training 

(Army;  now  called  SPAS) 

Job  Control  Number 

Job  Performance  Aid 

Job  Task  Performance  Test 
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LAS 

LRU 

LTTA 


Lockheed  Aircraft  Services 

Line-Replaceable  Unit 

Logic  Tree  Troubleshooting  Aid 


MACT 

MAINTIP 

MCAS 

MDC 

MDS 

MFHBF 

MIL  STD 

MIMS 

MITIPAC 

Mk/Mod 

MMTR 

MTTR 

MOS 

MTBF 

MTBR 

MTM 

MTS 

MTU 


Malfunction  and  Circuitry  Trainer 
Maintenance  Training  Improvement  Program  (NTEC) 
Marine  Corps  Air  Station 
Maintenance  Dependency  Chart 
Model/Designation/Series  (Air  Force) 

Mean  Flight  Hours  Between  Failures 
Military  Standard 

Maintenance  Instruction  Manual  System 

Modular  Integration  of  Training  Information  by  a 

Performance  Aiding  Computer  (Navy) 

Mark/Model 

Military  Manpower  Training  Report 
Mean  Time  to  Repair 

Military  Occupational  Specialty  (Army) 

Mean  Time  Between  Failures 
Mean  Time  Between  Repairs 
Maintenance  Training  Management 

Mobile  Training  Set  (for  Field  Training  Detachment) 
Maintenance  Training  Unit 


NALCOMIS 

NAMP 

NAMTD 

NARF 

NAS 

NAVAIR 

NEC 

NFE 

NIP 


Navy  Air  Logistics  Command  Management  Information 
System 

Naval  Aviation  Maintenance  Program 

Naval  Air  Maintenance  Training  Detachment 

Naval  Air  Rework  Facility 

Naval  Air  Station 

Naval  Air  Systems  Command 

Navy  Enlisted  Classification 

Not  Fully  Equipped 

NAMP  Improvement  Program 
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NOR 

NORM 

NORS 

NPRDC 

NSN 

NSRDC 

NTEC 

NTIPP 

NTIPS 

OJT 

O  level 

PACAF 

PE 

PIMO 

PINTO 

PMTRADE 

POMO 

PROM 

REES 

RFP 

ROM 

RTE 


SAC 

SAMT 

SDC 

SEL 

SIMMS 


Not  Operationally  Ready 

Not  Operationally  Ready  -  Maintenance 

Not  Operationally  Ready  -  Supply 

Navy  Personnel  Research  and  Development  Center 

National  Stock  Number 

David  W.  Taylor  Naval  Ship  Research  and  Development 
Center 

Naval  Training  Equipment  Center 

Navy  Technical  Information  Presentation  Program 

Navy  Technical  Information  Presentation  System 

On-the-Job  Training 
Organizational  Level  Maintenance 

Pacific  Air  Force 
Program  Element 

Presentation  of  Information  for  Maintenance  and 
Operation 

Performance  Improvement  for  Navy  Training  Organizations 
Program  Manager  for  Training  Devices  (Army) 
Production-Oriented  Maintenance  Organization  (AF) 
Programmable-Read-Only-Memory 

Reactive  Electronic  Equipment  Simulator 
Request  for  Proposal 
Read-only  Memory 

Resident  Training  Equipment  (for  technical  training 
center ) 

Support  Action  Code 

Simulated  Avionics  Maintenance  Trainer 
Sample  Data  Collection 
Selected  Equipment  List 

Symbolic  Integrated  Maintenance  Manual  System 


SMART 


System  Malfunction  Analysis  Reinforcement  Trainer 
SMTE  Simulated  Maintenance  Task  Environment 

SOJT  Supervised  On-the-job  Training 

SPAS  Skill  Performance  Aids  System  (Army;  previously  ITDT ) 

SQT  Skill  Qualification  Test  (Army) 

SRA  Shop-Replaceable  Assembly 

SRU  Shop-Replaceable  Unit 

STRES  Simulator  Training  Requirements  and  Effectiveness 

Study  ( AFHRL) 

TAG  Tactical  Air  Command 

TACAN  Tactical  Air  Navigation 

TAMMS  The  Army  Maintenance  Management  System 

TICCIT  Time-shared,  Interactive  Computer  Controlled  Informa¬ 

tion  Television 

TJS  Tactical  Jamming  System 

TM  Type  Maintenance 

TMS  Type/Model/Series  (Navy) 

TOT  Task-Oriented  Training 

TRAM  Target  Recognition  Attack  Multi-sensor 

TRU  Tester  Replaceable  Unit 

USAFE  United  States  Air  Force  -  Europe 

VAST  Versatile  Avionics  Shop  Test  (Navy) 

V T A S  Visual  Target  Av  iuisiticn  System 

WBS  Work  Breakdown  Structure 

WC  Work  Center 

WUC  Work  Unit  Code 
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Maintenance  and  Material  Management  System  (Navy) 


66-1  Air  Force  Maintenance  Management  System  (name 

&  66-5  derived  from  the  Air  Force  manual  that  sets  forth 
maintenance  policy.) 
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GLOSSARY 

Courseware:  Student  handbooks  and  manuals  and  that  port- ion  of 

the  set  of  computer  programs  resident  in  a  simulator  that 
implements  the  simulation  model  and  otherwise  addresses  the 
operation/functioning  of  the  equipment  being  simulated. 

Cross-Skill  Maintenance:  Maintenance  associated  with  one  skill 
area  that  is  performed  by  personnel  trained  in  a  different 
skill  area. 

Depot-Level  Maintenance:  Rear  area  maintenance,  major  repair 
or  equipment  modifications  performed  largely  by  civilians 
in  military  organizations. 

Direct-Support  Maintenance:  Intermediate-level  maintenance  per¬ 
formed  in  units  attached  to  or  organic  with  large  combat 
units,  e.g.,  divisions  (Army). 

Fidelity:  A  normative  term  that  describes  the  extent  to  which 

a  simulator  duplicates  its  operational  counterpart.  Physi¬ 
cal  fidelity  refers  to  physical  appearance,  since  a  simu¬ 
lator  may  be  two-dimensional  or  three-dimensional,  or  some 
combination,  in  its  construction.  Functional  fidelity 
refers  to  the  extent  to  which  the  performance  characteris¬ 
tics  of  operational  equipment  have  been  duplicated  in  the 
s imu lator . 

General  Support  Maintenance:  Intermediate-level  maintenance 
performed  in  units  attached  to  higher  commands,  e.g., 

Corps,  Theatre  forces. 

Intermediate-Level  Maintenance:  Maintenance  performed  in  a 
shop  by  a  maintenance  or  repair  unit. 
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Maintenance  Action:  All  effort  associated  with  the  completion 
of  a  maintenance  requirement  (e.g.,  the  correction  of  a 
malfunction)  that  permits  the  return  of  equipment  to  an 
operational  status. 

Maintenance  Task:  A  single  procedure  that  is  performed  as  part 

of  a  maintenance  action.  For  example,  remove,  troubleshoot, 
repair,  and  install  are  discrete  tasks  in  a  maintenance 
action  that  corrects  a  malfunction  and  restores  equipment 
to  operational  status. 

Off-Equipment  Maintenance:  Maintenance  performed  on  equipment 
systems  and  assemblies  that  have  been  removed  from  weapon 
end-  cems . 

On-Fquipment  Maintenance:  Maintenance  performed  on  equipment 
systems  and  assemblies  while  they  are  installed  on  weapon 
end- i terns , 

Organizational  Level  Maintenance:  Maintenance  performed  directly 
on  operational  equipment  (e.g.,  fault  detection,  component 
replacement)  by  personnel  assigned  to  units  that  operate 
the  equipment. 

Simulation:  The  imitative  representation  of  the  operation/ 

functioning  of  one  system  by  another  system.  It  consists 
of  the  simulation  model,  display  and  control  panels,  and 
other  input/output  facilities  peculiar  to  the  system  being 
s  imulated. 

Simulation  Model:  A  mathematical  model  that  describes  the 

operation/functioning  of  a  particular  system  or  equipment. 

Simulator:  The  device  (i.e.,  hardware  and  software)  on  which  a 

simulation  is  implemented. 

Simulator  Model:  Simulators  of  a  given  (complete  or  partial) 
con  figuration. 

Software:  That  portion  of  the  set  of  computer  programs  resi¬ 

dent  in  a  simulator  that  is  not  unique  or  peculiar  to  the 
system  being  simulated  (i.e.,  the  routines  concerned  with 
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utilities,  input/output ,  translation,  etc.  that  are  em 
ployed  for  general  control  of  the  computer). 

Team  Maintenance:  Maintenance  actions  or  tasks  that  are  pe 
formed  by  more  than  one  person. 
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